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Abstract 


This  report  documents  user  validation  of  the  Maritime  Visual  Analytics  Prototype  (MVAP),  a  test¬ 
bed  and  showcase  for  novel  visual  analytics  strategies  in  support  of  maritime  situation  awareness, 
anomaly  detection,  and  detailed  analyses  of  Vessels  of  Interest  (VOIs).  MVAP  components  can  be 
used  independently  or  integrated  into  Defence  Research  and  Development  Canada’s  (DRDC’s) 
Widget  Application  Shell  (WAS)  framework  in  the  Intelligence  Science  and  Technology  Integration 
Platform  (ISTIP). 

Validation  trials  were  done  with  three  objectives: 

•  Evaluate  the  usability  of  MVAP  apps  and  services 

•  Estimate  the  potential  impact  of  MVAP  on  MSOC  productivity 

•  Identify  potential  future  improvements  of  MVAP  components. 

A  “User  Jury”  methodology  was  used,  in  which  four  groups  of  Marine  Security  Operations  Center 
(MSOC)  analysts  (totaling  16  persons)  were  given  a  thirty  minute  hands-on  lesson  in  how  to  use 
MVAP,  and  then  challenged  to  achieve  six  new  tasks  in  fifteen  minutes  using  the  MVAP  tools.  The 
challenge  tasks  focused  on  maritime  domain  awareness  using  previously-recorded  real  data  from  the 
Global  Positioning  Warehouse  (GPW)  dataset. 

Trial  participants  reported  on  the  perceived  value  of  the  new  apps  using  the  internationally- 
recognized  System  Usability  Scale  (SUS).  Participants  were  observed  and  their  actions  were 
recorded  during  both  the  training  and  the  challenge  segments  of  the  trials,  with  particular  attention 
paid  to  successful  tasks,  errors  made,  and  self-learning.  Data  was  collected  through  observations, 
the  standard  SUS  questionnaire,  and  participants’  comments  at  a  Hot  Wash-Up  discussion.  The 
quality  of  the  apps  was  assessed  using  standard  SUS  scoring. 

End-user  response  to  the  trial  was  very  enthusiastic.  The  average  SUS  score  was  76/100  which  is 
considered  “Excellent”  in  the  scientific  literature.  Comments  from  participants  indicated  that  MVAP 
may  lead  to  a  significant  improvement  in  productivity  over  the  MSOC’s  current  tools,  and  the 
analysts  are  keen  to  have  it  on  their  desks.  The  Map  and  Timeline  app  generated  the  most 
comments  because  of  its  ability  to  visualize  the  time  evolution  of  the  maritime  picture,  and  because 
it  can  detect  close  encounters  between  vessels.  The  Magnets  Grid  app  also  drew  a  lot  of  praise 
because  it  offers  a  completely  new  way  to  explore  numerical  and  textual  attributes  of  vessels. 
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Ce  rapport  documente  la  validation  du  Prototype  d’Analytique  Visuelle  Maritime  (PAVM)  par  les 
usagers;  un  banc  d’essai  et  de  demonstration  pour  de  nouvelles  strategies  d’analytique  visuelle  en 
support  a  l’analyse  de  la  situation  maritime,  a  la  detection  d’anomalies  et  a  l’analyse  approfondie  de 
navires  d’interet.  Les  composantes  du  PAVM  peuvent  etre  utilisees  independamment  ou  integrees 
au  cadriciel  Widget  Application  Shell  (WAS)  de  Recherche  et  Developpement  pour  la  Defence 
Canada  (RDDC)  sur  la  plateforme  d’integration  Intelligence  Science  and  Technology  Integration 
Platform  (ISTIP). 

Les  essais  de  validation  ont  ete  realises  avec  trois  objectifs  : 

•  Evaluer  l’utilisabilite  des  apps  et  services  du  PAVM 

•  Estimer  T  impact  potentiel  du  PAVM  sur  la  productivite  des  Centres  des  operations  de  la  surete 
maritime  (COSM) 

•  Identifier  des  elements  d’ ameliorations  futures  potentielles  des  composantes  du  PAVM. 

Une  methodologie  “User  Jury”  a  ete  employee.  Quatre  groupes  d’analystes  du  Centre  des  operations 
de  la  surete  maritime  (COSM)  (pour  un  total  de  16  personnes)  ont  re<?u  une  formation  pratique  sur 
Tutilisation  du  PAVM  et  ont  ensuite  tente  d’accomplir  six  nouvelles  taches  en  quinze  minutes  en 
utilisant  les  outils  du  PAVM.  Les  taches  portaient  sur  la  connaissance  de  la  situation  maritime  et 
utilisaient  des  donnees  preenregistrees  reelles  provenant  de  la  base  de  donnees  Global  Positioning 
Warehouse  (GPW). 

Les  participants  de  l’essai  ont  evalue  la  valeur  perdue  des  nouvelles  apps  a  l’aide  du  System 
Usability  Scale  (SUS)  qui  est  reconnu  intemationalement.  Les  participants  ont  ete  observes  et  leurs 
actions  notes  pendant  le  segment  formation  autant  que  pour  la  partie  evaluation  des  essais,  avec  une 
attention  particuliere  portee  envers  les  taches  realisees  avec  succes,  les  erreurs  effectuees  et  l’auto- 
apprentissage.  Des  donnees  ont  ete  recoltees  a  l’aide  d’observations,  des  questionnaires  SUS 
standards  et  des  commentaires  des  participants  lors  de  la  discussion  apres  exercice.  La  qualite  des 
apps  a  ete  calculee  en  utilisant  le  pointage  standard  SUS. 

La  reception  du  PAVM  par  les  usagers  finaux  a  ete  tres  enthousiaste.  Le  pointage  SUS  moyen  etait 
de  76/100,  ce  qui  est  considere  «  Excellent »  dans  la  litterature  scientifique.  Les  commentaires  des 
participants  ont  indique  que  le  PAVM  promet  des  ameliorations  significatives  de  la  productivite  par 
rapport  aux  outils  actuels  du  COSM,  et  les  analystes  sont  desireux  de  T avoir  dans  leur 
environnement  de  travail.  L’app  Map  and  Timeline  est  celle  qui  a  genere  le  plus  de  commentaires 
en  raison  de  sa  capacite  de  visualiser  revolution  temporelle  de  la  situation  maritime  et  parce  qu’elle 
permet  de  detecter  les  rencontres  rapprochees  entre  des  navires.  L’app  Magnets  Grid  a  aussi  rcgu 
une  bonne  appreciation  car  elle  permet  d’exploiter  les  proprietes  numeriques  et  textuelles  liees  aux 
navires  d’une  maniere  completement  nouvelle. 
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Executive  summary 


Maritime  Analytics  Prototype  Development:  Validation  Final 
Report 

Eric  Hall;  Mike  Davenport;  Neil  Bozowsky;  William  Wright;  DRDC  Valcartier  TR 
[enter  number  only:  9999-999];  Defence  R&D  Canada  -  Valcartier;  November  c; 
Unclassified. 

Introduction  or  background: 

This  report  documents  user  trials  and  performance  assessments  in  Phase  3  of  a  Maritime  Visual 
Analytics  Prototype  (MVAP)  test-bed  implementation.  This  work  was  done  under  Advanced 
Research  Project  (ARP)  1  ljm  (previously  1  lhm)  “Maritime  Domain  Analysis  through  Collaborative 
and  Interactive  Visualization.”  It  builds  on  earlier  requirements  analysis,  literature  and  product 
survey,  design,  architecture,  and  implementation  contracts. 

This  research  specifically  examines  how  Visual  Analytics  can  help  surveillance  agencies  achieve 
better  maritime  domain  awareness  by  offering  cognitively  rich  representations,  information  filtering 
and  clutter  reduction  capabilities,  multimodal  interactions,  and  teamwork  collaboration  services.  To 
avoid  reproducing  already  well-established  and  widely  used  software  systems,  the  focus  is  on 
developing  compact  apps  (a  commonly-used  abbreviation  for  “applications”)  or  widgets  that  might 
be  incrementally  added  to  existing  surveillance  systems.  For  the  project,  Oculus  was  responsible  for 
project  management,  technical  architecture,  user  experience  design  refinement,  software 
development  and  demo  scenarios.  Salience  was  responsible  for  subject  matter  expertise,  analysis, 
design,  refinement,  demo  scenarios,  and  trials  data  synthesis. 

MVAP  was  implemented  using  a  Service  Oriented  Architecture  with  JavaScript  and  Java  code.  The 
following  apps  were  developed  and  tested:  Analysis  Sets,  Vessel  Summary  Card,  Record  Browser, 
Map  and  Timeline,  Magnets  Grid,  and  Events  Timeline. 

Methodology: 

An  early  onsite  planning  session  was  held  in  Halifax  ten  months  before  the  validation  event. 
Nineteen  MSOC  analysts  representing  the  Royal  Canadian  Navy  (RCN),  Coast  Guard,  RCMP,  and 
CORA  were  briefed  on  MVAP  capabilities  and  gave  suggestions  that  influenced  the  tasks  and 
datasets  for  the  validation. 

The  MVAP  apps  and  services  were  evaluated  to  determine  how  well  they  help  analysts’ 
effectiveness,  efficiency,  and  satisfaction.  This  was  measured  through  hands-on  User  Jury  trials 
with  the  generous  participation  of  personnel  from  the  Marine  Security  Operations  Center  (MSOC) 
in  Halifax.  Trial  data  included  data  recorded  at  the  MSOCs,  new  information  scraped  from  the  web, 
and  hand-crafted  or  computer-generated  data  to  support  scenarios  of  interest.  Subjects  were  given 
thirty  minutes  of  hands-on  training,  and  then  asked  to  perform  six  tasks  using  the  new  apps. 

Two  types  of  assessment  were  made: 

•  Observations  and  Anecdotes:  Interaction  with  the  new  apps  was  observed  during  both  the 
training  and  the  task-focused  halves  of  the  trial.  Usability  metrics  included  interaction  (e.g. 
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immediate  feedback),  errors,  critical  incidents  (e.g.  recovery  from  error)  and  self-learning  (i.e. 
discovering  capabilities  that  were  not  taught). 

•  Subjective  Assessments:  Participants  were  invited  to  give  subjective  assessments  of  perceived 
effectiveness,  efficiency,  correctness,  satisfaction,  and  trust  of  each  feature  using  the 
internationally  recognized  system  usability  scale  (SUS). 

Results: 

The  average  SUS  score  was  76/100,  which  corresponds  to  “excellent.” 

Verbal  reports  from  the  participants  were  enthusiastic  including  (paraphrased): 

•  “This  is  better  than  what  we  have  now.” 

•  “Very  useful  -  can’t  wait  to  use  it,  when  do  we  get  it?” 

•  “This  is  a  tool  that  an  analyst  would  use.” 

•  “The  GUI  was  intuitive  enough  that  participants  were  trying  features  without  waiting  for 
training” 

•  “All  apps  represent  concepts  that  would  greatly  improve  the  visual  analysis” 

In  their  comments,  participants  were  thinking  ahead  to  how  MVAP  could  be  integrated  into  MSOC. 
They  also  asked  whether  MVAP  could  also  be  used  in  a  context  disconnected  from  the  MSOC  data 
grid.  They  made  suggestions  for  data  exchange  with  existing  systems,  adding  configurability  to 
match  their  task-specific  requirements,  and  extending  the  capabilities  of  some  apps.  They  also 
made  suggestions  for  improving  the  user  interfaces. 

The  Map  and  Timeline  app  was  flagged  by  some  as  “the  best  app  of  all.”  Participants  appreciated 
how  it  visualized  time  evolution,  showed  vessels  interacting,  and  displayed  Close  Encounters 
information.  They  stated  that  it  is  almost  impossible  to  detect  close  encounters  with  the  tool  suites 
currently  at  the  MSOCs.  Magnets  Grid  was  also  popular,  in  part  because  it  is  surprisingly  simple  to 
use  and  provides  uniquely  valuable  insights. 

Documentation 

This  document  provides  full  documentation  of  the  trials  and  trial  results.  A  complete  description  of 
the  methodology  is  provided,  including  a  training  script  and  a  script  for  the  exercise.  The  original 
hand-written  comments  and  observations  are  reproduced  in  Annex  B,  and  all  the  comments  are 
assembled,  sorted,  and  interpreted  in  Tables  1  through  6. 

Future  plans: 

This  document  concludes  by  outlining  the  steps  required  to  deploy  the  MVAP  apps  to  an 
operational  environment. 
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1  Background  and  Functional  Context 


This  is  the  Validation  Report  of  the  Maritime  Visual  Analytics  Prototype  (MVAP)  development 
contract  W7701-135425  [17].  The  work  was  conducted  by  Oculus  Info  Inc.  and  Salience  Analytics 
Inc.  from  September  2012  through  December  2013. 

1.1  Context  and  Background 

The  MVAP  has  been  developed  to  address  specific  needs  of  the  MSOC  partners  and  the  Canadian 
Forces  in  protecting  the  maritime  approaches  to  Canada.  In  order  to  effectively  assess  the  value  of 
MVAP,  it  is  appropriate  to  first  summarize  what  those  needs  are,  and  highlight  how  Visual 
Analytics  can  contribute  to  meeting  them.  Section  1.1.1  thus  provides  a  summary  of  maritime 
surveillance  context  and  operational  challenges,  and  Section  1.1.2  introduces  the  government 
agencies  that  work  together  to  meet  those  challenges.  Section  1.1.3  provides  a  brief  introduction  to 
Visual  Analytics. 

Visual  Analytics  is  not  the  only  technology  that  is  potentially  relevant  to  Maritime  Domain 
Awareness,  and  DRDC  has  explored  many  other  strategies.  Sections  1.1.4  and  1.1.5  reference  some 
of  those  other  initiatives. 

1.1.1  Maritime  Domain  Awareness 

Maritime  domain  operators  and  analysts  around  the  world  have  a  mandate  to  achieve  Maritime 
Domain  Awareness,  which  is  defined  as  follows: 

•  “Maritime  Domain  Awareness  is  the  effective  understanding  of  everything  on,  under,  related 
to,  adjacent  to,  or  bordering  a  sea,  ocean  or  other  navigable  waterway,  including  all  maritime- 
related  activities,  infrastructure,  people,  cargo,  vessels,  or  other  conveyances.”  [16] 

This  mandate  is  based  on  the  need  to  protect  from  attack,  defend  sovereignty,  detect  illegal 
activities,  and  support  search  and  rescue  activities.  Operators  and  analysts  maintain  24/7  watch  over 
the  oceans  in  support  of  the  mandate.  They  do  so  by  extracting  and  analyzing  situational  facts  from 
a  variety  of  sensor  data  streams  and  analysis  tools. 

In  Canada,  the  Marine  Security  Operation  Centres  (MSOCs)  (see  Section  1.1.2)  rapidly  process  a 
variety  of  information  streams  in  order  to  develop  shared  situational  awareness,  which  is 
communicated  through  the  “Recognized  Maritime  Picture”  (RMP),  a  shared  multi-modal 
description  of  all  that  is  known.  Key  goals  of  this  analysis  are  to: 

•  Understand  what  the  observations  mean,  and  thus 

•  Anticipate  what  may  happen  next,  so  that  appropriate  action  may  be  taken  in  a  timely  manner. 

Such  insights  are  more  likely  to  be  achieved  if  the  MSOC  agencies  share  their  data  streams  and 
work  together  to  interpret  them. 

This  goal  of  rapid,  collaborative  extraction  of  meaning  from  high-volume  data  streams  is  very 
challenging  because  of: 
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•  The  large  number  and  diversity  of  data,  information  and  knowledge  types  and  sources; 

•  Significant  information  overload; 

•  The  incomplete  /  uncertain  nature  of  the  information;  and 

•  The  need  for  timeliness  in  the  resulting  RMP. 

1.1.2  Marine  Surveillance  Operational  Centres 

Responsibility  for  gathering  Maritime  information,  and  for  responding  to  actionable  information,  is 

divided  among  the  member  agencies  of  the  Marine  Surveillance  Operational  Centres  (MSOCs)  [11]. 

The  identities  and  roles  of  these  agencies  can  be  summarized  as  follows: 

•  Royal  Canadian  Navy  and  the  Canadian  Forces: 

♦  Surveillance  Role:  host  agency  for  the  Halifax  and  Esquimalt  MSOCs,  lead  agency  for 
assembling  the  global  Recognized  Maritime  Picture,  operator  of  the  CF  maritime 
surveillance  aircraft,  ships,  and  sensor  systems,  and  primary  point  of  contact  for 
exchanging  information  with  international  military  surveillance  agencies.  Maintains 
situational  awareness  throughout  Canada’s  area  of  responsibility  (which  extends 
beyond  Canadian  territorial  waters)  and  in  selected  regions  around  the  world,  such  as 
around  Canadian  warships.  CF  is  not  allowed  to  collect  information  on  Canadian 
citizens. 

♦  Response  Mandate:  primary  responder  for  events  in  international  waters,  for  events 
involving  a  foreign  warship,  and  for  any  maritime  act  of  war  against  Canada. 
Frequently  requested  to  support  other  MSOC  agencies  (e.g.  RCMP)  for  actions  within 
their  jurisdiction. 

•  Royal  Canadian  Mounted  Police  (RCMP): 

♦  Surveillance  Role:  host  agency  for  the  Great  Fakes  MSOC,  able  to  collect  some 
information  on  Canadian  citizens. 

♦  Response  Mandate:  primary  responder  for  law  enforcement  against  ships  in  Canadian 
territorial  waters,  including  the  Great  Fakes,  Atlantic,  Pacific,  and  Arctic  oceans. 

•  Department  of  Fisheries  and  Oceans  (DFO)  including  the  Canadian  Coast  Guard  (CCG): 

♦  Surveillance  Role:  CCG  maintains  situational  awareness  of  ship  movements  in 
Canadian  waters  through  its  Marine  Communications  and  Traffic  Services  (MCTS) 
operations  centers.  They  collect  call-in  data  from  ships  approaching  Canadian  ports, 
and  track  the  locations  of  the  ships  using  their  network  of  shore-based  radars  and 
Automatic  Identification  System  (AIS)  receivers.  DFO  flies  surveillance  aircraft, 
primarily  to  monitor  fishing  zones.  Both  DFO  and  CCG  operate  ships  that  provide 
surveillance  data.  DFO  has  access  to  vessel  tracking  system  (VTS)  data  from  fishing 
boats  in  or  near  Canadian  waters. 

♦  Response  Mandate:  CCG  enforces  conformance  to  shipping  regulations.  DFO 
enforces  conformance  to  fishing  regulations  and  restrictions.  Both  DFO  and  CCG  can 
intercept  and  board  ships  suspected  of  infringing  regulations. 

•  Transport  Canada  (TC): 
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♦  Surveillance  Role:  oversees  security  at  Canada’s  ports,  detects  marine  pollution 
events,  detects  regulatory  infractions,  regulates  commercial  shipping  in  Canadian 
waters,  regulates  security  at  offshore  facilities  such  as  oil  and  natural  gas  drilling 
platforms. 

♦  Response  Mandate:  imposes  regulations,  collects  evidence,  fines  or  prosecutes 
offenders,  educates  and  certifies  mariners,  shares  information  with  other  agencies. 

•  Canada  Border  Services  Agency  (CBSA): 

♦  Surveillance  Role:  collects  information  about  cargo  and  passengers  destined  for 
Canada,  before  they  leave  a  foreign  port.  They  maintain  a  network  of  intelligence 
agents  in  Canada  and  abroad  to  detect  threats  to  Canada's  immigration,  visitor,  refugee 
and  citizenship  programs,  and  to  detect  contraband  products. 

♦  Response  Mandate:  ability  to  block  entry  of  persons  or  cargo  into  Canada.  Provide 
intelligence  reports  that  can  lead  to  interventions  by  the  RCN  or  RCMP. 

1.1.3  Visual  Analytics 

A  widely  accepted  summary  of  Visual  Analytics  is: 

•  “Visual  analytics  is  the  science  of  analytical  reasoning  facilitated  by  interactive  visual 
interfaces.  People  use  visual  analytics  tools  and  techniques  to  synthesize  information  and 
derive  insight  from  massive,  dynamic,  ambiguous  and  often  conflicting  data;  detect  the 
expected  and  discover  the  unexpected;  provide  timely,  defensible,  and  understandable 
assessments;  and  communicate  assessment  effectively  for  action”  (Page  4  of  [21]). 

Visual  analytics  is  thus  a  multidisciplinary  field  that  includes  the  following  focus  areas: 

•  Analytical  reasoning  techniques  that  let  users  obtain  deep  insights  that  directly  support 
assessment,  planning,  and  decision  making; 

•  Visual  representations  and  interaction  techniques  that  exploit  the  human  eye’s  broad 
bandwidth  pathway  into  the  mind  to  let  users  see,  explore,  and  understand  large  amounts  of 
information  simultaneously; 

•  Data  representations  and  transformations  that  convert  all  types  of  conflicting  and  dynamic  data 
in  ways  that  support  visualization  and  analysis; 

•  Techniques  to  support  production,  presentation,  and  dissemination  of  analytical  results  to 
communicate  information  in  the  appropriate  context  to  a  variety  of  audiences. 

Thus  we  can  expect  that  visual  analytics  may  significantly  improve  the  Recognized  Maritime 
Picture  by  offering  cognitively  rich  representations,  information  filtering  and  clutter  reduction 
capabilities,  multimodal  interactions  and  teamwork  collaboration  services,  in  order  to  provide  better 
insight  into  information  and  increased  situation  awareness. 

1.1.4  Related  DRDC  Projects 

This  contract  builds  on  foundational  work  done  by  Defence  Research  and  Development  Canada 
(DRDC)  in  the  following  fields: 
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•  Information  visualization  and  management  for  enhanced  domain  awareness  in  maritime 
security  (Project  1  lhe). 

•  Collaborative  Knowledge  Exploitation  for  Maritime  Domain  Awareness  (Project  1  lhg). 

•  Multi-hypothesis  Link  Analysis  for  Anomaly  Detection  in  the  Maritime  Domain  (Project 
llhk). 

1.1.5  Previous  Work  under  ARP  1 1jm/1 1  hm 

This  contract  is  part  of  a  series  that  have  been  funded  under  DRDC’s  “Maritime  Domain  Analysis 
through  Collaborative  and  Interactive  Visualization”  project  ARP  lljm  (previously  llhm),  also 
known  as  Design  of  a  Maritime  Visual  Analytics  Prototype  (DMVAP).  Previous  contracts  under 
this  project  include: 

•  Literature  and  product  survey  [6,  8]; 

•  MSOC  (Marine  Security  Operations  Centre)  site  visits  and  requirements  analysis  [5]; 

•  Design  study  for  the  prototype  [10]; 

•  Architecture  study  for  the  prototype  [2]; 

•  Scenario  descriptions  and  dataset  requirements  analysis  [9]; 

•  Reports  on  the  development  and  testing  of  the  core  applications  [12] 

1 .2  Project  Objectives 

The  over-reaching  objective  of  this  project  and  this  contract  is: 

a.  Develop  VA  and  collaborative  technologies  that  help  the  Canadian  Forces  and  MSOC 
personnel  achieve  their  maritime  surveillance  mandate  (Section  1.1.1). 

Such  technologies  aim  to  achieve  the  following  [18]: 

b.  Support  rapid  awareness  and  identification  of  key  information  elements  and  thus  rapid 
insight  into  the  meaning  of  a  situation; 

c.  Improve  the  visualization  of  the  recognized  maritime  picture; 

d.  Enable  better  comprehension  of  a  situation  and  how  it  could  develop; 

e.  Reveal  what  is  not  known  (data  gaps  and  uncertainty); 

f.  Support  detection,  alerting,  and  visualization  of  anomalies; 

g.  Enable  collaborative  team  work. 

In  order  for  the  technologies  to  address  objective  (a)  the  following  project  objectives  were 
addressed,  as  described  in  Section  7: 

h.  Communicate  to  operational  units  the  benefits  of  technologies  developed  in  the  project; 

i.  Identify  a  mechanism  to  transition  the  technology  from  research  to  operational  systems; 
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The  specific  objectives  of  this  contract  are  (  Annex  A  page  1  of  [17]): 

j.  Design,  develop,  and  test  how  the  Phase  3  Maritime  Visual  Analytics  Prototype 
(MVAP)  supports  Maritime  Domain  Awareness  (MDA).  This  Phase  3  portion  of  the 
work  follows  on  from  Phases  1  and  2  that  were  documented  previously  [12]; 

k.  Perform  validation  activities  (usability  trials)  for  all  the  visual  apps  and  widgets  of  the 
MVAP  to  estimate  how  well  objectives  (b)  though  (g)  have  been  met.  Sections  2 
through  6  of  this  document  focus  on  the  validation  methodology  and  results. 

The  Oculus  proposal  [15]  suggested  that  validation  of  the  apps  might  address  the  following 
objectives,  as  discussed  in  Section  7: 

l.  Establish  rigorous  evidence  that  can  be  used  to  promote  the  MVAP  approach  to 
operational  users,  and  thus  facilitate  transition  of  the  technology  to  them. 

m.  Establish  the  relevance  of  MVAP  technology  to  a  wider  set  of  users  than  the  MSOCs 
and  the  RJOCs. 

n.  Extract  scientifically  rigorous  observations  suitable  for  publication  in  the  scientific 
literature. 


5 


2  Methodology  and  Metrics 


The  MVAP  apps  developed  in  this  project  addressed  operational  objectives  (b),  (c),  (d)  and  (f). 
Therefore  Task  2,  Prototype  Apps  Validation,  focused  on  how  well  those  four  objectives  were 
addressed  in  the  apps.  Section  2.1  outlines  the  software  attributes  that  are  important  to  achieve 
these  objectives. 

Sections  2.2  and  2.3  describe  the  selected  validation  methodology,  which  includes  training, 
exercises,  user  juries,  and  structured  interviews. 

Section  2.4  identifies  three  types  of  metrics  that  will  be  used  in  the  assessment,  and  Section  2.5 
introduces  the  System  Usability  Scale  questionnaires  that  were  used. 

2.1  Operational  Performance  Goals 

The  U.S.  Department  of  Health  &  Human  Services  describes  the  impact  to  be  expected  from  well- 
designed  software  [22],  Their  description  includes: 

•  Increased  ease  of  use 

•  Increased  ease  of  learning 

•  Increased  user  satisfaction 

•  Increased  trust  in  the  system 

•  Faster  accomplishment  of  tasks 

•  Reduced  number  of  user  errors 

•  Reduced  need  for  documentation 

ISO  9641-10  [13]  provides  the  following  principles  of  good  human-computer  interaction: 

•  Suitability  for  the  task 

•  Self-descriptiveness  of  the  action 

•  Controllability 

•  Conformity  with  user  expectations  or  consistency 

•  Error  tolerance 

•  Suitability  for  learning. 

ISO  9641-11  [13]  similarly  summarizes  the  attributes  of  good  software  as  how  well  a  product  can  be 
used  to  achieve  specified  goals,  in  a  specified  context  of  use  (users,  tasks,  equipments  and 
environments).  Good  software  will  achieve: 

•  Effectiveness  (task  completion  by  users) 

•  Efficiency  (quickly  and  accurately) 

•  Satisfaction  (good  user  experience) 
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O’Connell  and  Choong  [14]  also  identify  the  following  criteria  for  effective  human-computer 
interactions: 

•  Responsiveness  (e.g.  immediate  feedback) 

•  Minimal  actions  (i.e.  interaction  efficiency) 

Scholtz  [20]  proposes  that  these  lists  must  be  extended  for  Visual  Analytics  software  in  a  number  of 
directions.  Of  most  interest  on  this  project  is  the  suggestion  that  VA  software  should  contribute  to 
Situation  Awareness,  meaning  it  should  help  analysts: 

•  Perceive  essential  information 

•  Interpret  it  correctly 

•  Use  it  to  predict  what  will  happen  next. 

2.2  Methodology 

The  validation  methodology  was  a  User  Jury  and  included:  training  of  MSOC  analysts,  operational 
tasking  within  the  context  of  a  scenario,  and  assessment  of  software  performance  against  the  above 
performance  goals.  The  full  methodology  can  be  summarized  as  follows: 

1.  Planning:  An  early  onsite  planning  session  was  held  on  January  10th  and  1 1th  2013  in  Halifax. 
Nineteen  MSOC  analysts,  representing  the  RCN,  Coast  Guard,  RCMP,  and  CORA  attended, 
watched  a  presentation  of  the  MVAP  developments,  provided  suggestions  for  the  validation 
trials,  and  helped  establish  usability  and  operational  performance  goals. 

2.  Experimental  Design:  The  validation  methodology  (this  document)  was  developed  by  the 
Oculus  team  and  reviewed  by  clients  (DRDC)  and  operational  staff.  The  methodology  included: 

♦  Identification  of  validation  metrics  (see  Section  2.3). 

♦  Definition  of  Operational  Use  Cases  (see  Section  4.1). 

♦  Development  of  appropriate  datasets. 

♦  Design  of  a  testing  timeline,  as  shown  in  Figure  1 . 

♦  Installation  of  MVAP  apps  on  multiple  workstations. 

♦  Observer  training  on-site  before  the  trials  (see  Section  5.1). 

3.  Participants:  Participants  were  all  MSOC  analysts,  already  familiar  with  the  goals  and 
challenges  of  Maritime  Domain  Awareness.  Participants  were  separated  into  groups  of  up  to  5 
people,  with  each  group  spending  approximately  45  minutes  being  trained,  running  the  trial 
task,  and  providing  feedback  as  discussed  below. 

4.  Participant  Training:  Each  group  of  participants  was  given  30  minutes  of  hands-on  training  by 
Oculus,  as  described  in  Section  3. 

5.  Scenario-Based  Exercise:  Each  participant  undertook  an  exercise  that  lasted  about  15  minutes, 
to  find  information  in  the  trial  dataset  using  the  MVAP  apps  and  fill  out  the  worksheet  shown  in 
Annex  A,  as  described  in  Section  4.  All  participants  were  given  the  same  exercise. 
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6.  Observation:  MVAP  team  members  observed  the  participants  during  training  and  during  the 
exercise,  taking  notes  using  the  Observer  Form  as  discussed  in  Section  5. 

7.  Subjective  Reports:  At  the  end  of  the  exercise,  participants  filled  out  an  SUS  Questionnaire 
(see  Section  2.5  and  Annex  A)  and  a  Ranking  Survey.  Scores  were  assigned  and  interpreted  as 
described  in  Section  2.5.2. 

8.  Hot  Wash-Up:  After  all  participants  had  handed  in  their  written  notes,  there  was  a  brief  “Hot 
wash-Up”  discussion  with  all  participants.  Comments  were  noted  by  the  observers. 

9.  Synthesis:  The  observations  and  surveys  were  assembled  and  interpreted  as  described  in 
Section  6. 


Participants 


Observers 


Example 

Elapsed 

Time 

(hours) 


Observer 

Training 


0  -\ 


CL 
13 

o 

i  -I  o 


2  -J 


3  -I 


Participant  Training 


Scenario-Based  Exercise 


Subjective  Reports 
&  Hot  Washup 


CM 

CL 

o 

O 


Participant  Training 


Scenario-Based  Exercise 


Subjective  Reports 
&  Hot  Washup 


Do 

Training 

Observations 


Hot  Washup  and 
Documentation 


Reset  Workstations 


Do 

Training 

Observations 


Hot  Washup  and 
Documentation 


Figure  1:  Timeline  of  the  Methodology* 

Participants  are  trained  for  30  minutes,  then  conduct  an  exercise  for  30  minutes,  then  answer  a 

questionnaire  about  the  VMAP  apps. 


2.3  Participant  Equipment 

Participants  were  given  the  following  equipment: 

•  MS  Windows  7  laptop  computer  with  network  access  to  a  common  server. 

•  Google  Chrome  browser, 

•  A  printed  sheet  describing  the  exercise  assignment,  with  space  for  results  to  be  documented,  as 
described  in  Section  4. 

•  Blank  assessment  sheets,  for  usability  assessment  as  described  in  Section  2.5. 

All  computers  used  the  following  resources  on  a  shared  server: 

•  Apache  Tomcat  server, 

•  MVAP  software  running  on  the  server, 

•  PostgreSQL  database,  containing  all  training  data  and  all  exercise  data. 


2.4  Metrics 

Four  types  of  metric  were  collected  for  each  participant: 

•  Observations  and  Anecdotes:  Trial  observers  watched  the  participants,  identified  the  type 
and  number  of  non-critical  errors  made  (e.g.  false  starts,  requests  for  help,  etc.)  and  recorded 
them  on  the  Observer  Worksheet  in  Annex  A.  Participants  were  also  invited  to  record 
unstructured  suggestions,  and  these  were  collected  with  the  observer  notes. 

•  Time  on  Task:  An  attempt  was  made  to  have  the  observers  record  the  elapsed  time  for  each 
task  milestone  [24]  using  the  Observer  Worksheets.  After  the  first  trial,  the  observers  agreed 
that  it  was  not  possible  to  accurately  estimate  those  times,  so  this  initiative  was  abandoned. 

•  App  Ranking:  Participants  ranked,  on  a  scale  of  0  to  2,  how  valuable  each  app  would  be  if 
integrated  into  the  domain  awareness  systems  that  they  use. 

•  System  Usability  Scale:  Participants  provided  their  subjective  assessments  of  the  apps  using 
the  System  Usability  Scale  questionnaires  provided  in  Section  2.5. 

2.5  System  Usability  Scale 

The  System  Usability  Scale  was  proposed  by  Brooke  [3,  23]  as  a  subjective  assessment  of  usability 
that  is  general  enough  to  be  widely  applicable  across  domains.  Sauro  [19]  provides  a  helpful 
discussion  of  how  to  interpret  SUS  results. 

Although  the  SUS  questions  do  not  directly  address  all  of  the  performance  goals  in  Section  2.1,  we 
have  avoided  customizing  them  because: 

•  Brooke’s  standard  list  of  questions  is  a  universal  benchmark  for  usability. 

•  The  SUS  list  of  10  questions  was  created  very  carefully,  by  first  generating  50  questions, 
running  trials,  and  then  down-selecting  those  questions  where  differing  opinions  emerged  most 
clearly  [4]. 
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•  The  questions  are  also  arranged  carefully,  so  that  for  example  the  positive  statements  such  as 
“I  liked. . are  alternated  with  negative  statements  such  as  “I  did  not  like  ...” 

Annex  A  lists  the  SUS  questionnaires  for  all  the  MVAP  apps.  Brooke  provides  a  template  for  the 
SUS,  and  gives  permission  for  it  to  be  freely  re-used  on  the  condition  that  any  resulting  publications 
acknowledge  its  source  (page  6  of  [4]). 

2.5.1  How  the  Questionnaires  were  Used 

In  accordance  with  Brooke’s  instructions  [4],  the  questionnaires  were  used  as  follows: 

•  A  SUS  questionnaire  was  handed  to  each  participant  after  the  Scenario-Based  Exercise,  but 
before  any  debriefing  or  discussion  took  place. 

•  Participants  were  instructed  to  record  their  immediate  response  to  each  item,  rather  than 
thinking  about  items  for  a  long  time. 

•  Participants  were  instructed  to  insert  a  response  for  every  item.  If  a  respondent  felt  unable  to 
respond  to  a  particular  item,  they  were  to  mark  the  centre  point  of  the  scale. 

2.5.2  How  the  Questionnaires  were  Scored 

Each  questionnaire  yielded  a  single  number  representing  a  composite  measure  of  the  overall 
usability  of  the  app  being  studied.  Scores  for  individual  items  on  a  questionnaire  are  not  meaningful 
alone. 

The  SUS  score  for  each  questionnaire  was  calculated  as  described  below,  and  as  shown  in  Figure  2: 

•  The  “scale  position”  for  a  question  is  the  small  number  under  the  checked  box. 

•  For  odd-numbered  questions  (1,  3,  5,  7,  and  9)  calculate  the  score  contribution  as  the  scale 
position  minus  1,  and  write  that  to  the  right  of  the  scale. 

•  For  even-numbered  questions  (2,  4,  6,  8,  and  10)  calculate  the  score  contribution  as  5  minus 
the  scale  position,  and  write  that  to  the  right  of  the  scale. 

•  Confirm  that  all  the  score  contributions  range  from  0  to  4. 

•  Add  all  the  score  contributions  and  then  multiply  by  2.5  to  get  an  overall  value  that  ranges 
from  0  to  100. 

Based  on  statistics  published  by  Sauro  [19],  the  SUS  results  can  be  translated  into  a  percentile 
values.  A  percentile  value  of  75%,  for  example,  means  that  the  MAP  scored  higher  than  75%  of  the 
usability  studies  done  by  Sauro. 
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Mark  one  box  in  each  row. 


System 


1 .  1  think  that  I  would  like  to  use  these  apps 
frequently 


2.  I  found  the  apps  unnecessarily  complex 


3.  1  thought  the  apps  were  easy  to  use 


4.  1  think  that  I  would  need  the  support  of  a 
technical  person  to  be  able  to  use  these  apps 


5.  1  found  the  various  functions  in  these  apps 
were  well  integrated 


6.  1  thought  there  was  too  much  inconsistency 
in  these  apps 


7.  I  would  imagine  that  most  people  would 
learn  to  use  these  apps  very  quickly 


8.  I  found  the  apps  very  cumbersome  to  use 


9.  I  felt  very  confident  using  these  apps 


10. 1  needed  to  learn  a  lot  of  things  before  I 
could  get  going  with  these  apps 
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of  a  questionnaire 


3  Training  the  Participants 


Oculus  experts  trained  each  group  of  participants  as  part  of  the  software  validation  process,  as 
shown  in  Figure  1.  Time  allocated  for  training  was  strictly  limited  to  30  minutes,  and  addressed 
each  relevant  MVAP  app  in  turn,  as  described  in  Sections  3.1  through  3.6. 

3.1  Analysis  Set  Manager 

Time  for  this  training:  5  minutes. 

Procedure: 

•  Open  a  Chrome  browser  and  go  to  the  MVAP  bookmark 

(http : / / <server> : <port>/MVAP) 

•  Click  on  the  Analysis  Set  Manager 

•  Within  the  Analysis  Set  Manager,  click  the  Add  Group  button  and  create  a  set  called  1RCS 

♦  Select  set  IRCS 

♦  Click  Add  Ship 

♦  Click  “+”  to  add  a  search  criterion 

♦  Specify  that  criterion  as  “Name”,  “Contains”,  “IRCS”  and  click  “Search” 

♦  Toggle  vessel  selection  by  clicking  on  Vessels  and  “OK.”  This  adds  the  vessels  to  set 
IRCS 

•  Right  click  on  IRCS  BAREU00  and  remove  the  vessel  from  set  IRCS 

•  Create  another  set  called  IRCS  6 

•  Cut  and  paste  ship  IRCS  BATEU6  from  IRCS  to  IRCS  6 

3.2  Map  and  Timeline 

Time  for  this  training:  6  minutes. 

•  Click  the  Map  button  to  launch  the  map  and  timeline  for  IRCS  6 

♦  Place  the  map  cursor  over  Brussels  and  use  the  mouse  wheel  to  zoom  in 

♦  Place  the  cursor  over  the  upper  timeline  and  use  the  mouse  wheel  to  zoom  in 

♦  Click  on  the  orange  time  window  handles  and  drag  them  to  enclose  November  6th 

♦  Drag  the  white  region  on  the  timeline  to  animate  around  Nov6 

♦  Return  to  the  Analysis  Set  Manager  and  delete  both  sets  IRCS  6  and  IRCS. 

•  Launch  Map  and  Timeline  with  no  set  selected 

♦  Zoom  the  map  to  Halifax  and  the  Bay  of  Fundy 

♦  Zoom  the  timeline  to  July  10-12  201 1 
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♦  Click  the  Fetch  Vessels  button 

♦  Animate  vessel  movement  by  dragging  the  white  time  region 

♦  Mouse  over  vessels  to  see  their  names 

♦  Select  ship  “Carrier”  and  observe  that  a  Close  Encounter  pop-up  appears 

♦  Double  click  Carrier  to  see  details  then  close  the  resulting  window 

♦  Click  export  and  put  all  vessels  in  a  new  set  called  “Fundy  ROF’ 

♦  Return  to  the  analysis  set  manager  and  launch  Record  Browser  then  Magnets  Grid  on 
Fundy  ROI 

3.3  Record  Browser 

Time  for  this  training:  5  minutes 

•  In  the  Analysis  Set  Manager,  highlight  the  “Fundy  ROI”  set  and  then  click  on  the  “launch 
record  browser”  button 

♦  Observe  the  data  items  on  each  summary  card:  vessel  image,  NATO  icon,  Flag, 
24h/96h  indicators,  VOI  markers,  word  cloud. 

♦  Flip  the  card  to  see  the  expanded  list  of  vessel  properties. 

♦  Click  on  the  grey  bar  to  “select”  a  card  and  add  it  to  the  export  button 

♦  Drag  the  card  to  the  left  to  compare  it  to  other  cards. 

♦  Select  a  second  card. 

♦  Click  the  “Add  Analysis  Set”  button,  name  the  new  set,  click  “OK”,  and  note  that  the 
new  set  is  visible  in  the  Analysis  Set  Manager 

3.4  Magnets  Grid 

Time  for  this  training:  5  minutes 

•  In  the  Analysis  Set  Manager,  select  “Fundy  ROI”  and  launch  the  Magnets  Grid  app 

♦  Observe  that  vessels  are  represented  as  dots  (“dust”) 

♦  Click  the  “Shake”  button  to  push  dots  apart  randomly. 

♦  Mouse  over  a  dot  to  see  vessel  information 

•  Create  magnets  for  Vessel  length,  Speed,  and  Net  Tonnage 

♦  Place  the  magnets  and  shake  each  one  to  see  how  the  dust  is  attracted 

•  Click  “Add  Calculated  Magnet” 

♦  In  the  Calculated  Magnets  dialog,  specify  a  new  magnet  as  “Country  =  Jacardia” 

♦  Place  the  new  magnet,  shake  it,  and  observe  three  ships  “Carrier”,  “CarpeDiem”,  and 
“Gigaloo”  are  drawn  toward  that  magnet 
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♦  Use  marquee  drag  to  select  those  three  ships,  then  export  them  as  a  new  Analysis  Set 
called  “Fundy  VOI” 

3.5  Timeline 

Time  for  this  training:  3  minutes 

•  In  the  Analysis  Set  Manager,  select  “Fundy  VOI”  and  launch  the  Timeline  app 

•  Use  the  timeline  to  visualize  events  for  these  three  ships. 

•  Double-click  on  a  name-change  event  to  synchronize  multiple  timelines. 

•  Split  Gigaloo  and  compare  Jun  12  to  May  1. 

3.6  GPW  Map  and  Timeline 

Time  for  this  training:  4  minutes 

•  Explain  how  GPW  Map  and  Timeline  differs  from  Map  and  Timeline 

•  Launch  the  GPW  Map  and  Timeline 

♦  Zoom  in  on  Sable  Island 

♦  Zoom  timeline  to  January  1,2012  00:00-03:00 

♦  Fetch  all  vessels  near  Sable  Island 

♦  Select  ship  “Panuke  Sea”,  get  compressed  tracks,  and  encounters 

♦  Follow  the  Panuke  Sea  track  to  find  an  encounter  south  of  St.  John’s 

♦  Animate  the  interaction  by  sliding  the  time  slider. 
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4  Scenario-Based  Exercise 


If  the  Training  session  (Section  3)  achieved  its  ambitious  timeline,  two  minutes  remained  to  instruct 
the  participants  in  the  Scenario-Based  Exercise,  and  then  15  minutes  for  them  to  complete  that 
exercise.  The  exercise  proceeded  as  follows: 

•  Participants  and  observers  checked  that  their  computers  were  in  an  acceptable  state  for  the 
exercise 

•  Oculus  then  assigned  the  exercise  as  described  in  Section  4.1 

•  Participants  were  given  a  worksheet  (Annex  B)  that  included: 

♦  A  summary  of  the  assigned  exercise. 

♦  A  set  of  fill-in-the-blank  questions  to  be  answered  during  the  exercise. 

•  Observers  watched  how  the  participants  used  the  MVAP  software  to  achieve  the  exercise 
goals,  as  described  in  Section  5 

♦  If  participants  asked  “how  to”  questions,  the  observers  answered  them  verbally  but  did 
not  touch  the  computers.  Such  events  were  recorded  by  the  observers. 

•  When  all  participants  had  achieved  the  objectives  (usually  after  about  15  minutes)  Oculus 
collected  the  worksheets  and  handed  out  the  “Ranking”  sheets  for  the  observers  to  fill  out. 

4.1  Scenario  for  the  Exercise 

•  Intercepted  signals  suggest  that  a  person  of  interest  (Mr  X)  was  on  or  near  the  Bahamas  on 
April  1st,  2012. 

♦  Task  1 :  use  the  GPW  to  find  all  vessels  near  the  Bahamas  on  that  day. 

•  A  signal  from  a  different  person  of  interest  (POI  2)  was  intercepted  around  that  time  that 
included  the  word  “Carnival.”  Intelligence  analysts  suspect  a  possible  collaboration  between 
Mr  X  and  POI  2,  and  hence  need  to  know  what  ships  with  that  name  were  at  or  near  the 
Bahamas  on  that  day. 

♦  Task  2:  find  the  track  of  such  a  vessel  whose  name  includes  the  word  “Carnival.” 

•  If  Mr  X  was  on  a  Carnival  ship,  we  are  interested  in  knowing  the  next  port  of  call  of  that  ship, 
arrival  time,  and  route. 

♦  Task  3:  Animate  the  track  of  the  selected  vessel  to  find  their  next  port  of  call  and  time 
of  arrival,  and  to  determine  if  the  route  was  direct  or  indirect. 

♦  Task  4:  Export  the  vessels  and  tracks  to  the  MVAP,  and  create  a  “Carnival”  analysis 
set. 

♦  Note:  for  the  actual  trial,  there  was  no  internet  connectivity.  Thus,  it  would  not 
have  been  possible  to  retrieve  vessel  images  and  meta  data  from  the  internet.  So, 
the  vessels  were  exported  prior  to  the  trial  and  stored  in  the  database.  During  the 
trial,  users  simply  opened  the  analysis  set  manager  and  searched  for  vessels 
named  Carnival. 
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Information  indicates  Mr  X  may  be  on  one  of  the  newer,  larger  Carnival  ships,  we  need  to 
characterize  the  tonnage  and  build  year  of  each  vessel. 

♦  Task  5:  Use  the  Magnets  Grid  to  characterize  the  ships  using  a  combination  of  Build 
Year  and  Net  Tonnage. 

The  vessel  of  interest  is  flying  under  a  Panamanian  flag. 

♦  Task  6:  Use  the  Record  Browser  to  compare  the  ship  flags. 


5  Procedures  for  Observers 


The  role  of  the  observers  was  to  notice  and  record  behaviours  of  the  participants  that  revealed 
software  flaws  and  software  strengths. 

5.1  Preparation 

Observers  received  two  types  of  training: 

•  MVAP  Training:  in  order  to  recognize  how  participants  are  interacting  with  the  apps,  the 
observers  first  gained  familiarity  with  the  apps.  This  was  required,  for  example,  in  case  a 
participant  used  MVAP  in  an  unexpected  way  -  an  untrained  observer  might  mark  this  as  an 
error,  whereas  a  trained  observer  might  recognize  that  the  participant  has  found  a  better 
strategy.  This  requirement  was  addressed  by  having  the  observers  first  execute  the  usability 
trials,  on  the  morning  of  the  experiments,  as  shown  in  the  Figure  1  timeline. 

•  Usability  Evidence  Training:  observers  were  taught  how  to  observe  user  behaviours  and 
evidence  for  usability,  and  how  to  recognize  events  of  interest,  using  guidelines  from  the 
literature  as  described  in  Section  5.2. 

5.2  Observing  Participants  during  Training  and  Tasked 
Activities 

Observer  used  one  worksheet  (see  Annex  A)  for  each  participant  to  record  relevant  actions  and 
events  as  the  participants  learned  to  use  the  MVAP  apps  and  as  they  used  the  apps  to  tackle  the 
assigned  tasks.  An  attempt  was  made  to  record  the  elapsed  time  for  each  assigned  task,  but  these 
values  were  deemed  unreliable  and  hence  were  not  used.  The  observers  watched  for  the  following 
specific  types  of  actions  and  events: 

•  Critical  Errors:  Critical  errors  are  deviations  that  prevent  the  participant  from  finishing  a 
task.  Participants  may  or  may  not  be  aware  that  the  task  goal  is  incorrect  or  incomplete.  [24] 

•  Non-Critical  Errors:  Non-critical  errors  are  errors  that  are  recovered  by  the  participant  and 
do  not  block  the  successful  completion  of  the  task.  These  errors  result  in  the  task  being 
completed  less  efficiently.  For  example,  exploratory  behaviors  such  as  opening  the  wrong 
navigation  menu  item  or  using  a  control  incorrectly  are  non-critical  errors.  [24] 

•  Unanticipated  Strategies:  Participants  may  discover  ways  to  use  the  apps  that  were  not 
anticipated  during  the  design  process.  Such  observations  are  helpful  because  they  identify 
previously  unrecognized  app  capabilities  that  may  benefit  from  further  development. 

•  Self-Learning:  Observers  may  observe  participants  initially  failing  to  solve  a  problem,  and 
then  organically  discovering  (i.e.  without  being  taught)  the  capability  of  the  apps.  Such 
observations  are  helpful  because  they  identify  strengths  in  the  apps  that  need  to  be  protected. 
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5.3  Hot  Wash-Ups 

The  observers  led  the  Hot  Wash-Up  discussions  and  recorded  all  the  responses  and  comments  made 
by  the  participants. 
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6  Detailed  Assessment  of  Results 


Sixteen  MSOC  analysts  and  specialists,  in  four  groups,  participated  in  user  trials  November  13th  and 
14th  2013  in  Halifax.  Analyst  affiliations  included: 

•  Royal  Canadian  Navy 

•  RCMP 

•  Transport  Canada 

•  Canadian  Border  Services  Agency 

•  Department  of  Fisheries  and  Oceans 

•  Coast  Guard 

The  trials  were  conducted  as  described  in  Sections  3  through  5.  Annex  B  contains  the  raw  hand¬ 
written  notes  from  these  sessions. 

The  comments  and  observations  from  these  trials  are  reported  and  analysed  in  the  following 
subsections.  Tables  1  through  6  list  every  received  comment,  cross-referenced  to  Annex  B.  Where 
many  similar  comments  were  received,  a  single  summary  is  recorded  in  the  table  but  cross- 
references  are  provided  to  all  the  original  written  comments.  Thus  the  presence  of  multiple  cross- 
references  is  evidence  that  an  opinion  was  widely  held  among  the  participants. 

All  the  observers  attended  the  same  Hot  Washup  meetings,  so  there  is  a  lot  of  overlap  in  their 
reports  of  those  meetings.  When  a  Hot  Washup  comment  is  inserted  into  a  table,  only  one  cross- 
reference  is  provided  because  only  one  comment  was  made,  even  if  that  comment  was  reported  by 
multiple  observers. 

6.1  Overall  Assessment  of  the  MVAP 

Table  1  encapsulates  comments  about  the  MVAP  as  a  whole,  including  System  Usability  Survey 
(SUS)  results.  The  SUS  results  averaged  76/100,  which  is  better  than  75%  of  new  software  systems 
[19].  Figure  3  shows  average  scores  for  each  SUS  topic.  Table  8  in  Bangor  et  al  [1]  shows  that 
76/100  corresponds  to  a  rating  of  “Excellent”  which  is  the  second-highest  rating  on  their  scale. 
Participants  in  the  comments  used  words  similar  to  “excellent”  to  describe  the  MVAP  apps. 

Written  comments  from  the  participants  were  similarly  positive,  suggesting  that  MVAP  capability 
would  be  a  valued  asset  and  certainly  a  step-up  from  the  systems  currently  being  used  at  the  MSOC. 
A  number  of  commenters  encouraged  us  to  explore  using  MVAP  more  widely  than  the  MSOC. 

Recommendations  from  Table  lean  be  summarized  as  follows  (in  no  particular  order): 

a.  Deploy  MVAP  to  the  MSOC  as  soon  as  possible 

b.  Find  a  way  to  also  deploy  MVAP  for  operations  that  are  isolated  from  the  MSOC 
databases 
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c.  Extend  MVAP  to  read  and  write  a  wider  variety  of  file  formats  and  types,  including 
Alerts 

d.  Provide  the  ability  to  save  an  MVAP  session  to  file  for  sharing  or  later  resumption  of 
analysis  in  context 


Strongly  ,, 
Agree 


4H 


Strongly 

Disagree 


2  H 


1  -l 


1 . 1  think  that  I  would  like  to  use  these  apps  frequently 

7. 1  would  imagine  that  most  people  would  learn  to  use  these  apps  very  quickly 

3. 1  thought  the  apps  were  easy  to  use  9.1  felt  very  confident  using  these  apd 

5. 1  found  the  various  functions  in  these  apps  were  well  integrated 


10. 1  needed  to  learn  a  lot  of  things  before  I  could  get  going  with  these  apps 

2. 1  found  the  apps  unnecessarily  complex 

4. 1  think  that  I  would  need  the  support  of  a  technical  person  to  be  able  to  use  these  apps 


8. 1  found  the  apps  very  cumbersome  to  use 

6. 1  thought  there  was  too  much  inconsistency  in  these  apps 


Figure  3  Average  Scores  on  the  Ten  SUS  Survey  Questions 
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—  Analysis  Set  Manager 


--  Map  and  Timeline 


--  Record  Browser  and  Vessel  Summary  Cards  --  Magnets  Grid 


--  Events  Timeline 


Figure  4  Average  Ranking  of  the  Five  Apps 


Table  1  Comments  and  Observations  about  MVAP  in  General 


Maritime  Visual  Analytics  Prototype 

System  Usability  Survey  (SUS): 

•  Number  of  SUS  surveys  handed  in:  14 

•  Scores:  average  =  76,  median  =  79,  lowest  =  40,  highest  =  92 

•  Percentile:  MVAP  scored  better  than  75%  of  new  software  releases  (using  statistics  from  [19]) 

Positive  Comments  not  Specific  to  an  App: 

MVAP  had  Immediate  Appeal: 

•  The  “buzz  on  the  street”  is  that  MVAP  is  pretty  cool  [C. 1.4. la] 

•  Between  other  systems  or  MVAP,  this  is  the  best  [C.3.2.4a] 

•  Very  useful  -  can’t  wait  to  use  it,  when  do  we  get  it?  [C.2.3.2d,  C.3.4.1b,  C. 3. 1.4a] 

•  This  is  a  tool  that  an  analyst  would  use.  [C.3.2.1a] 

GUI  was  Intuitive: 

•  GUI  was  intuitive  so  participants  were  trying  features  without  waiting  for  training  [C.1.1.4h] 

MVAP  would  Add  Value  in  Various  Contexts: 

•  All  apps  represent  concepts  that  would  greatly  improve  the  visual  analysis  [C.2.4.2a,  C.3.4.1d] 

•  We  would  be  interested  in  using  this  tool  isolated  from  the  big  databases  [C.3.3.3k] 

•  This  should  be  in  MSOC,  not  just  the  RCN  watch  floor  [C.3.2.3c] 

•  This  would  be  useful  for  link  analysis  [C.3.4.1a] 

Negative  Comments  not  Specific  to  an  App: 

•  There  were  bugs  apparent  when  trying  to  Drag  and  Drop  [C.2.1 ,2f] 

•  Integration  between  the  apps  should  be  improved  [C.2.4.2b,  C.3.4.1e,  3. 4. 2d] 

Suggestions  for  the  Future: 

•  Need  more  time  to  evaluate  these  apps  properly  [C.2.2.5a] 

Improve  the  GUI: 

•  Support  “right-clicking”  on  visual  elements  to  get  a  listing  of  options  [C.2.3.2b] 

•  Provide  MVAP  as  a  toolbox  on  the  analyst  dashboard,  where  each  tool  can  clicked  when  needed  [C.3.2.3f] 
Improve  data  sharing: 

•  Support  import  of  other  formats  of  data,  including  alert  results  [C.  1.3. la,  C.  1.1. 4b,  C.3.1.1n,  C.3.3.3j] 

•  Export  analysis  results  to  Greenline  or  Google  Earth  and  add  pretty  labels  [C.3.2.3b,  C.3.2.3n,  C.3.3.2g] 
Support  session  endurance  and  sharing: 

•  Provide  metadata  so  that  the  inferred  “analysis  story”  is  traceable  [C. 3. 1.1k] 

•  Add  the  ability  to  save  a  working  session,  share  with  collaborators,  or  come  back  to  it  at  a  later  time  [C.3.2.3a] 
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6.2  Assessment  of  the  Analysis  Set  Manager 


Table  2  presents  comments  that  were  specifically  about  the  Analysis  Set  Manager  (ASM).  On  the 
Participant  Ranking  Sheets  (see  Annex  A)  where  the  scale  ranges  from  0  “not  useful”  to  2  “very 
useful”,  the  average  ranking  of  the  Analysis  Set  Manager  was  1.9. 

Feedback  from  participants  was  positive,  and  included  suggestions  for  streamlining  the  user 
interface.  Participants  were  quick  to  understand  the  ASM  and  start  using  it  to  create  manageable 
groups  (“analysis  sets”)  of  contacts.  Initial  problems  with  the  user  interface  were  rare  and  can  be 
attributed  to  the  learning  curve  rather  than  design  flaws.  The  collected  observations  may  however 
point  to  simple  GUI  improvements  that  would  make  the  ASM  more  accessible.  As  stated  in 
Section  6.1,  participants  were  keen  to  extend  the  ASM  to  read  a  write  a  wider  range  of  data  formats 
and  to  share  analysis  sets  with  other  analysts. 

Not  only  did  participants  embrace  the  ASM’s  key  role  of  structuring  analysis  data,  they  were  keen 
to  extend  and  enhance  it.  They  recommended  that  work  be  done  to  define  better  analysis  set 
hierarchies  for  operational  analysts  to  use  with  the  ASM. 

Recommendations  from  Table  2  can  thus  be  summarized  as  follows  (in  no  particular  order): 

e.  Extend  ASM  to  read  and  write  a  wider  variety  of  file  formats  and  types,  including 
Alerts. 

f.  Tweak  the  ASM  GUI  to  improve  navigation,  selecting  ships  and  setting  up  a  search. 

g.  Define  a  deeper  structure  for  the  analysis  sets,  and  provide  the  ability  to  save  ASM  sets 
for  sharing  or  for  later  resumption  of  analysis. 
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Table  2  Comments  and  Observations  about  the  Analysis  Set  Manager 

Analysis  Set  Manager 

Average  User  Ranking:  1.9 
Positive  Comments: 

•  Clear  and  easy  to  understand  [C.2.3.3a,  C.2.3.1a,  C.2.4.3a 

•  Great  for  grouping  certain  datasets  [C. 2. 1.3a,  C. 2. 1.2a 

Evidence  of  Self-Learning: 

•  (None  reported) 

Suggested  New  Ways  to  Use  This  App: 

•  Export  sets  to  Excel  (the  app  already  supports  this)  [C.3.3.3n] 

Noted  Bugs  or  Criticisms: 

•  Clicking  MVAP  shortcut  does  not  take  you  back  to  the  Analysis  Set  Manager  (have  to  click  ASM 
button)  [C. 1.4.3b] 

•  Creating  folders,  and  adding  to  them,  is  non-trivial  [C.1 ,2.1b] 

Observed  Errors  When  Learning: 

Problems  setting  up  a  search: 

•  Clicked  “OK”  rather  than  “Search”  to  get  the  list  [C.1 ,2.4d,  C.1. 3. 1b  ,  C.1. 3. 2a,  C.1. 1.1  a  ,  C.I.I.I.f] 

•  Search  did  not  work  using  filter  “carnival”  instead  of  “Carnival”  [C.1.3.4g] 

Problems  selecting  a  sub-list  of  ships: 

•  Opened  the  Record  Browser  with  only  one  ship  instead  of  a  group  [C.1 ,2.4f,  C.1 ,2.4h] 

•  Used  “shift-click”  to  select  the  vessels  [C.1 .4.3a] 

•  Tried  clicking  the  first  in  the  Analysis  Set,  then  shift-clicking  the  last,  to  “Select  All”  [C.1. 1.1a] 

•  After  a  search  created  a  short-list,  did  not  select  the  desired  vessels,  just  clicked  “OK”  [C.1 .1 .3f] 

Suggested  Improvements  or  Added  Features: 

Improve  data  sharing: 

•  Ability  to  share  analysis  sets  between  analysts  or  with  other  systems  (e.g.  Greenline)  [C. 2. 1.1a,  C.1 ,2.1a, 

C. 1.2.3c,  C.3.1.3k,  C.3.2. 3m] 

•  Ability  to  import  a  vessel  set  from  a  non-MSOC  source  or  a  VMS  pull  from  a  fisheries  analysis  [C.1. 3. la, 
3.1.4o] 

•  Export  the  analysis  context  so  that  other  analysts  can  go  there.  [C. 3. 1.3k] 

Add  deeper  structure  to  the  analysis  sets: 

•  Save  analysis  sets  in  a  relevant  directory  structure  [C. 3. 1.3k] 

•  Each  problem  is  different  so  we  need  a  flexible  analysis  set  hierarchy  encoded  as  directories  or  groups  -  like  a 
recipe  [C.3.1.4n] 

Improve  the  GUI: 

•  Provide  more  efficient  method  to  select  multiple  ships  (use  standardized  GUI  method)  rather  than  having  to 
select  each  ship  [C. 2. 1.4a,  C.2.3.2a] 
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6.3  Assessment  of  the  Record  Browser  and  Vessel  Summary 
Cards 

Table  3  presents  the  comments  that  were  specifically  about  the  Record  Browser  and  Vessel 

Summary  Cards.  On  the  Participant  Ranking  Sheets  (see  Annex  A)  the  average  ranking  was  1.7. 

There  were  two  major  concerns  with  the  Record  Browser  and  Vessel  Summary  Cards: 

•  Two  participants  mentioned  that  software  is  already  available  to  provide  a  similar 
visualization. 

•  The  visualization  is  highly  dependent  on  having  a  rich  field  of  “tombstone”  facts  to  place  on 
the  back  of  the  card,  and  detailed  dynamic  information  (e.g.  recent  ports  visited)  to  place  on 
the  front  of  the  card. 

There  were  also  a  number  of  good  suggestions  about  how  to  improve  the  Record  Browser  and 

Vessel  Summary  Cards,  as  shown  in  Table  3.  Key  recommendations  are  (in  no  particular  order): 

h.  People  immediately  understood  the  Vessel  Summary  Card  concept  and  wanted  to 
extend  it  to  be  configurable  for  each  analysis  focus. 

i.  If  there  are  existing  apps  at  the  MSOC  that  provide  a  similar  Record  Browser 
functionality,  work  should  be  done  to  clarify  the  role  of  the  Record  Browser  when 
integrated  with  those  existing  apps. 

j.  Connect  the  vessel  summary  cards  (“Vessel  Summary  cards”)  “live”  to  the  database  so 
that  they  update  as  new  information  becomes  available. 
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Table  3  Comments  and  Observations  about  the  Record  Browser  and  Vessel  Summary  Cards 

Record  Browser  and  Vessel  Summary  Cards 
Average  User  Ranking:  1.7 
Positive  Comments: 

•  “Vessel  Summary  Card”  concept  is  useful  and  easy  to  understand  [C.2.3.3b,  C. 1.1. 3c,  C. 2. 1.3b] 

•  The  Word  Cloud  of  most-visited  ports  is  good  [C.2.4.3b] 

Evidence  of  Self-Learning: 

•  Opened  the  Record  Browser  without  a  set  selected,  then  figured  out  what  the  problem  was,  went  back  at 
selected  a  set  as  required.  [C.  1.3. 3a] 

•  When  the  graphic  national  flag  was  not  recognized,  flipped  the  card  to  read  the  country  name  [C.l.l.lh] 

•  Intuitively  dragged  a  card  to  the  left  display  position,  before  being  taught  this  [C.1 .4.3c] 

Suggested  New  Ways  to  Use  This  App: 

•  This  will  be  good  for  the  Watches  [C. 2. 1.4b] 

Noted  Bugs  or  Criticisms: 

•  This  capability  is  easily  available  in  other  software  [C.2.3.1b,  C.2.4.3b] 

•  Online  data  (used  for  the  prototype  Vessel  Summary  Cards)  is  often  out  of  date  [C.2.1 ,2b] 

Problems  with  the  GUI: 

•  Drag-and-drop  error  (?)  caused  two  transparent  cards  to  overlay  each  other  in  the  browser  [C.1 ,2.3f,  C.1 ,2.5b, 
C.1.1.4j,  C. 3.1.2a] 

•  The  user  tried  to  put  the  left  card  back  into  the  deck.  [C.1. 1.1  .d] 

•  Cumbersome  [C.2.1. 3b] 

Observed  Errors  When  Learning: 

•  Tried  to  mouse-over  the  flag  to  see  the  country  name  pop  up,  and  did  not  think  of  flipping  card  [C.1.2.3i, 
C.1.2.5e,  C.1. 1.4,  C.1. 1.4m] 

•  Double-clicked  on  the  analysis  set  and  got  only  a  single  card  [C.1.1.4d] 

Suggested  Improvements  or  Added  Features: 

Show  richer  information  fields: 

•  Allow  the  user  to  configure  the  Vessel  Summary  Card  contents  [C.2.1. 1b,  C.3.1.3j] 

•  The  card  should  update  automatically  as  the  underlying  data  changes  [C.2.1. 2b,  C.1. 1.3a] 

•  Vessel  Summary  Card  should  show  the  most  recent  port  [C.1 ,2.4b] 

Reveal  information  more  clearly: 

•  Provide  a  legend  or  tooltips  to  explain  the  info  shown  on  the  cards  [C.2.4.1a] 

•  Make  is  possible  to  see  top  and  flip-side  of  a  Vessel  Summary  Card  at  the  same  time  [C.1 ,2.5a] 
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6.4  Assessment  of  Magnets  Grid 

Table  4  presents  the  comments  that  were  specifically  about  the  Magnets  Grid  app.  On  the 
Participant  Ranking  Sheets  (see  Annex  A)  Magnets  Grid  scored  an  average  ranking  of  1.7. 

Magnets  Grid  created  a  significant  “buzz”  among  the  participants,  apparently  because  of  its  novelty, 
its  apparent  “abstract”  visualization  strategy,  and  its  very  quick  learning  curve.  Comments  such  as 
“awesome”  were  quite  common.  Some  participants  found  the  prototype  very  interesting  but  could 
not  see  how  it  would  be  useful  to  them,  in  part  because  it  is  most  useful  when  there  is  a  rich  array  of 
operationally-relevant  data  for  every  vessel,  and  they  did  not  expect  to  have  such  data  available. 

There  were  some  good  suggestions  about  how  to  improve  Magnets  Grid.  Some  of  these  are  just 
small  bugs  and  are  listed  in  Table  4  but  the  following  recommendations  merit  more  careful 
attention: 

k.  Magnets  Grid  is  surprisingly  simple  to  use  (surprising  because  it  is  difficult  to  explain 
verbally,  but  easy  to  understand  when  implemented)  and  provides  uniquely  valuable 
insights.  It  should  be  used  in  the  MSOCs  for  comparing  vessels,  multi-parameter 
searching,  and  fuzzy  analysis. 

l.  The  ability  to  create  new  rules  and  associate  them  with  magnets  is  very  good,  but  the 
software  should  make  that  process  more  efficient  and  user  friendly,  for  example  by 
using  more  pull-down  menus  to  select  data  fields. 

m.  The  Magnets  Grid  concept  should  be  linked  to  MSOC’s  trip  wires,  choke  points,  and 
regions  of  interest.  Thus  for  example  “hours  since  entering  Cabot  Straight”  might  be 
assigned  to  a  magnet. 
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Table  4  Comments  and  Observations  About  the  Magnets  Grid 

Magnets  Grid 

Average  User  Ranking:  1.7 
Positive  Comments: 

•  Interesting  /  awesome  representation  [C.2.3.1c,  C.2.4.3c,  Cl. 3. 4c,  C.3.3.3g,  C.3.2.1d] 

•  Very  useful  [C.2.1 .1c,  C.2.2.2a,  C.2.2.4a,  C.2.4.1b,  C. 3.1. 3d,  3.2.3m] 

•  Like  how  you  can  drag  and  select  and  export  to  an  analysis  set.  [C.2.1 ,2c,  C.3.3.3i] 

•  Much  better  than  reviewing  text  attributes  line-by-line  [C.3.2.3d] 

Evidence  of  Self-Learning: 

•  Figured  out  how  to  select  a  group  of  ships  and  then  export  them  as  an  Analysis  Set  [C.1 ,2.5g] 

•  Tried  out  various  magnet  arrangements  and  explained  what  the  patterns  meant  [C.1 ,2.5h] 

Suggested  New  Ways  to  Use  This  App: 

•  Use  this  to  compare  vessels  [C.2.4.3c] 

•  Look  for  newer  builds,  flag,  country  of  registry  [C.3.3.3g] 

•  Quite  practical  for  “fuzzy  analysis”  [C.2.1. 4c] 

•  To  view  discrete  events,  associate  the  event  types  with  the  colour  of  the  dots  [C.3.3.3h] 

Noted  Bugs  or  Criticisms: 

•  Usefulness  depends  on  getting  access  to  the  data  [C.2.1. Id,  C.2.1. 4c,  C.3.1.4f] 

•  Not  sure  when  I  would  use  this  [C.2.1. 3c,  C.2.3.1c] 

Conceptually  Difficult: 

•  Not  clear  what  the  arrows  mean  [C.3.2.3q] 

Software  Bugs: 

•  The  app  exported  “all”  instead  of  just  what  was  selected. [C.I.I.I.e] 

•  It  is  difficult  to  shake  the  magnets  [C.l.l.li] 

•  Accidently  got  a  blue  “select”  square  and  could  not  make  it  go  away  [C.1. 1.4k,  C.3.3.1a] 

Outliers  may  Dominate: 

•  When  one  vessel  is  an  outlier,  all  the  others  stay  tightly  clustered  (“Panuke  Sea  problem”)  [C.3.3.2a] 

Observed  Errors  When  Learning: 

•  Selected  “Length”  magnet  rather  than  “tonnage”  [C.1 ,2.4g] 

•  Double  clicked  from  Magnets  Grid  (MG)  and  lost  MG  context  and  did  not  see  that  a  new  tab  was  created 
[C.1.2.4i] 

•  Opened  two  ships,  found  in  MG,  directly  to  individual  cards  rather  than  to  an  analysis  set  [C.1 .3.2c] 

•  When  exporting  selected  vessels,  did  not  click  the  Radio  Button  [C.1.1.4e] 

Suggested  Improvements  or  Added  Features: 

•  Explain  what  it  means  to  have  a  magnet  representing  an  address  [C.1.3.4d] 

•  When  defining  magnets,  provide  pull-downs  on  record  fields,  so  you  don’t  have  to  remember  the  names 
[C.3.1 .3e] 

•  Provide  Magnets  Grid  calculators  that  use  geographic  measures  such  as  trip  wires,  choke  points,  Regions  of 
Interest  (ROIs),  boundary  area  [C.3.2.3o] 
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6.5  Assessment  of  Map  and  Timeline 

Table  5  presents  the  comments  that  were  specifically  about  the  Map  and  Timeline  app.  On  the 
Participant  Ranking  Sheets  (see  Annex  A)  Map  and  Timeline  achieved  an  average  ranking  of  1.9. 

The  Map  and  Timeline  app  generated  so  much  interest  that  Table  5  is  more  than  twice  as  large  as 
the  other  tables  in  this  Section.  A  number  of  people  said  that  this  is  the  best  app  of  the  MVAP  suite, 
and  much  better  than  other  systems  available  to  them  for  viewing  vessel  behaviour.  Participants 
were  particularly  excited  about  the  GPW  Map  and  Timeline,  which  uses  real  data  and  can  be  used  to 
report  on  close  encounters.  Participants  went  beyond  our  suggested  applications  to  suggest  that  the 
app  could  be  used  to  characterize  what  is  normal  (i.e.  “baseline  activity”)  and  to  communicate 
analytical  results  more  clearly  (as  a  video)  when  giving  a  briefing. 

The  ability  to  slide  backward  and  forward  in  time  was  very  popular  but  participants  found  the  time 
slider  user  interface  difficult  to  master.  Thirteen  separate  references  in  Table  5  were  directed  at  this 
problem.  When  it  was  suggested  that  this  might  just  be  a  learning  curve  issue,  an  MSOC  leader 
underscored  that  learning  curves  are  to  be  avoided  if  possible  because  of  the  high  tum-over  in 
manpower  at  the  MSOCs.  Participants  suggested  some  relatively  easy  solutions,  such  as  double¬ 
clicking  on  the  timeline  to  specify  a  desired  time,  or  providing  a  simple  pop-up  query  box. 

Given  the  high  value  accorded  to  the  Map  and  Timeline  information,  the  participants  wanted  more 
tools  to  help  access  that  data  efficiently.  Specific  recommendations  include: 

n.  Map  and  Timeline  is  excellent  and  important:  deploy  it  soon  as  a  primary  domain 
awareness  tool 

o.  Deploy  an  initial  version  of  Close  Encounters,  and  then  continue  to  improve  it  as  noted 
in  Table  5 

p.  Improve  the  timeline: 

■  Provide  more  standard  interactions  (e.g.  type  in  the  start  and  end  date  and  time) 

■  Make  the  graphic  timeline  more  user-friendly  and  robust  (e.g.  provide  tooltips  and 
improve  robustness). 

q.  Improve  the  robustness  of  the  track  animation: 

■  Prevent  situations  where  the  slider  can  be  moved  without  causing  the  tracks  to 
animate. 

r.  Add  more  features  to  the  Close  Encounters  tool: 

■  Provide  a  flexible  temporal  and  spatial  search  box 

■  Support  the  use  of  double-click  on  a  close  encounter  event  to  jump  both  the 
timeline  and  the  map  to  that  event 

■  Estimate  the  probability  that  a  close  encounter  was  not  an  accident 

■  Export  the  list  of  close  encounters 
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Table  5  Comments  and  Observations  About  the  Map  and  Timeline 

Map  and  Timeline 

Average  User  Ranking:  1.9 
Positive  Comments: 

•  This  is  the  best  app  of  all  [C. 2.1. 2d,  C.2.2.2a,  C.3.2.3g] 

•  Very  good  representation  [C.2.3.1d] 

•  Much  better  than  other  systems  where  it  is  almost  impossible  to  detect  a  close  encounter  [C.3.4.1c] 

•  Timeline  GUI  is  good  for  fine-grain  adjustments  when  zoomed  in  [C.2.4.1c] 

The  app  is  useful: 

•  Like  visualizing  time  evolution  [C. 2.1. 2d,  C. 2. 1.3d,  C.2.3.3c] 

•  Like  visualizing  multiple  vessels  interacting  [C.2.3.1d,  C.3.3.3d,  C.3.3.2d,  C.3.4.1c] 

•  Like  seeing  encounter  icons  directly  on  the  map  [C. 3. 1.1a] 

•  Like  being  able  to  search  for  events  (e.g.  close  encounters)  [C. 3.1 .3b] 

•  Like  being  able  to  communicate  vessel  tracks  [C.2.3.3c] 

The  app  uses  real  data: 

•  Like  this  app  if  it  will  handle  large  data  [C. 3.1 .3a] 

•  Like  how  you  can  import  tracks  (e.g.  Vessel  Management  System  /  Fishing)  as  spreadsheets  [C. 3.1 ,3m] 

Evidence  of  Self-Learning: 

•  Found  the  small  track  bars  in  the  lower  timeline  [C.1 ,2.5f] 

•  Figured  out  how  to  animate  on  the  map  using  one  handle  of  the  timeline  [C.l.l.lg] 

•  Tried  using  other  tools  to  find  Carnival  ships  [C.1.1.4i] 

Suggested  New  Ways  to  Use  This  App: 

Use  it  for  visualizing  motion: 

•  Useful  for  pattern-of-life  analysis  and  determining  baseline  activity  [C.2.4.3d] 

•  When  analysing  a  “play,”  use  the  timeline  to  animate  the  story,  then  hit  a  button  to  “play”.  [C.3.3.3b] 

•  Use  the  sliding  timelines  to  detect  polluters  [3.2.3r] 

•  For  example:  start  with  a  set  of  50  fishing  vessels,  ask  what  planes  flew  over  them,  ask  the  same  question  to 
another  agency,  then  bring  back  all  that  you  learned  and  put  it  in  the  tool.  [C.3. 1 . 1  j] 

Use  it  for  close  encounters  detection: 

•  Search  for  close  encounters  should  be  done  in  two  ways:  a)  “who  encountered  ship  N?”  and  b)  “of  all  ships 
within  a  Region  of  Interest,  where  did  close  encounters  occur?”  [C. 3. 1.3c,  C. 3. 1.4b] 

•  Automate  the  very  complex  analytical  process  at  the  MSOC  of  detecting  close  encounters  [C.2.4.3f] 
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Noted  Bugs  or  Criticisms: 

The  Timeline  interface  was  difficult  to  learn: 

•  Timeline  Bar  is  difficult  to  use  [C.2.1.4d,  C.2.1.3d,  C. 1.3.2b,  C.1.2.4j,  C.3.1.3f,  C.3.2.3f,  C.3.4.1k] 

•  Can’t  easily  tell  what  year  it  is  [C.1.1 ,2a,  C.1.1.3g] 

•  Sometimes  it  aligns  with  the  date  on  the  left  wall  of  the  window,  so  when  you  zoom,  the  desired  date  goes  out 
of  view  [C.1.1. 1b] 

•  If  the  white  area  in  the  timeline  is  too  wide  there  may  be  no  way  to  regain  control  of  the  slider.  [C.2.4.1c, 
C.1.3.4e,  C.3.1.3i] 

•  Tools  should  be  immediately  intuitive  to  avoid  the  need  for  a  learning  curve  because  there  are  many  analysts 
coming  and  going  [C.3.1.3h] 

It  was  difficult  to  animate  some  tracks: 

•  It  is  difficult  to  animate  the  tracks  [C.1 .3.4e,  C.3.1 .3i] 

•  If  the  white  area  in  the  timeline  is  too  wide,  or  if  the  tracks  end  within  the  time  interval,  the  tracks  no  longer 
animate  [C. 1.1. 1c,  C.1. 1.2d,  C.1. 3.4e,  C.3.1. 3i] 

•  Time  line  slider  is  jumpy  when  looking  at  Panuke  [C.1 ,2.5c] 

There  was  some  difficulty  fetching,  searching,  selecting,  and  exporting  ships  using  this  app: 

•  Not  clear  which  contacts  are  fetched  (all  within  the  go  area  and  time  window?)  [C.1.1 ,2b] 

•  Need  a  better  way  to  search  for  vessels  (“search  button”)  [C.1. 2. 3d,  C.1. 1.4a] 

•  Having  to  select  vessels  one  at  a  time  is  slow  [C.1. 2. 3d,  C.1.1.2e] 

•  Not  clear  whether  we  can  export,  to  the  Analysis  Set,  a  subset  of  ships  visible  in  the  Map  and  Timeline,  or 
whether  we  have  to  export  all  [C.1 .3.4b] 

•  Problems  dragging  a  vessel  into  the  Vessel  Summary  Card  Browser  [C.1.1.2h] 

There  was  some  confusion  between  the  Map  and  Timeline  and  the  GPW  viewer: 

•  Not  clear  whether  to  use  Map  and  Timeline  or  GPW  [C.1. 1.2c] 

•  The  app  runs  slow  on  the  test  platform  and  thus  was  difficult  to  evaluate  [C.2.2.5a] 

The  appearance  of  the  app  could  be  improved: 

•  It’s  hard  to  see  the  light  blue  tracks  against  the  grey  background  [C.1. 3. 4a] 

•  When  you  fetch  “encounters”  (which  takes  a  long  time)  the  GUI  freezes  -  should  have  “busy”  signal  [C.1 .1 .3b] 

•  Not  clear  what  Close  Encounters  plot  shows  [C.1 .4.4a] 

Observed  Errors  When  Learning: 

Errors  using  the  Timeline: 

•  Problems  zooming  and  panning  on  the  timeline  [C.1. 2. 1c,  C.1.2.4e,  C.1. 3. 1c,  C.1. 1.4c,  C.1. 1.4b] 

•  Selected  the  wrong  year  and  then  the  vessel  search  did  not  work  [C.1.2.4f,  C.1. 4. 1b] 

•  When  trying  to  go  to  a  future  port,  participant  scrolled  backwards  in  time  [C.1 ,2.3g 

•  Trouble  finding  date-  time  when  ship  is  in  port  (needed  to  animate  the  ship  to  port,  then  mouse-over) 

[C.1 .2.3. h] 

Errors  extracting  subsets: 

•  Trouble  creating  an  Analysis  Set  from  selected  vessels  [C.1.1 ,2g] 

•  Tried  to  select  all  the  Carnival  ships  at  once,  in  the  map  view  [C.1 ,2.5d,  C.1.3.4h,  C.1.1 ,2f] 

•  Confused  between  general  “fetch  tracks”  and  “compressed  tracks”  using  time  boundary  [C.l.l.li] 

Difficulties  with  GPW: 

•  Accidently  made  a  GPW  query  that  took  a  long  time  [C.1.1.4fg,  C.1. 4. 1c] 

•  Gave  up  after  a  long  delay  and  hit  “cancel”  on  GPW  Fetch  [C.1 .1 ,3e] 
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Suggested  Improvements  or  Added  Features: 

Improve  the  Timeline: 

•  Provide  an  alternative  where  you  type  in  dates  rather  than  scroll  the  slider  [C.2.3.2c,  C.3.1 .3h,  C.3.3.3a, 
C.3.2.3h,  C.3.2.3i,  C. 2.1. If] 

•  For  example  double-click  to  enter  a  date  to  snap  to  [C.3.4.2g] 

•  Maybe  use  a  calendar  pop-up  to  select  the  time  window  [C.3.2.3j] 

•  Double-click  on  the  timeline  to  go  to  that  date  [C.1 ,2.4a 

•  Use  iPhone-like  time  selector:  scroll  up  on  dials  to  define  the  time  [C.3.1 ,3g] 

Improve  the  visual  appearance: 

•  Provide  better  maps  and  map  perspectives  [C.2.1.1e,  C.3.3.3m] 

Improve  the  Close  Encounters  interface: 

•  Double-click  on  a  track  in  the  Close  Encounters  window  to  auto-zoom  to  the  time  of  close  encounter  and  to  the 
other  ship  [C.1 ,2.3a,  C.1 ,2.4c 

•  Allow  users  to  configure  the  spatial  and  temporal  width  of  the  close-encounters  box  [C.3.2.2a,  C.3.3.2h, 

C. 3.4. If] 

•  Create  a  textual  list  of  all  detected  Close  Encounters  [C.3.4.H] 

•  Can  we  assign  a  “probability  of  close  encounter”  as  a  percentage?  For  example  if  a  ship  stops,  that  increases 
the  chances  it  was  a  close  encounter  [C.3.4. 1  j] 

Improve  selection  of  ship  subsets: 

•  “Select  AH”  visible  ships,  then  export  them  as  an  Analysis  Set  to  be  searched  by  attribute  [C.1.3.4f,  C.1.1.2f, 
C.3.4.1e] 

•  Provide  a  search-on-feature  capability  to  select  ships  on  the  map  [C.3.4. 1g] 

Provide  enhanced  output  products: 

•  Export  an  animated  track  as  a  video,  perhaps  to  play  in  a  separate  widget  [C.3.3.3c,  C.3.3.2c] 
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6.6  Assessment  of  the  Events  Timeline 


Table  6  presents  the  comments  that  were  specifically  about  the  Events  Timeline  app.  On  the 
Participant  Ranking  Sheets  (see  Annex  A)  this  app  scored  an  average  ranking  of  1.5,  which  is  the 
lowest  score  of  the  five  apps  tested.  This  app  was  taught  last,  when  time  was  tight,  and  it  was  not 
required  for  the  challenge  tasks,  which  may  explain  this  low  score. 

The  Events  Timeline  uses  a  timeline  with  similar  controls  to  the  Map  and  Timeline  app,  and  thus 
suffered  from  some  of  the  same  learning-curve  issues.  Developers  should  ensure  that  the  Events 
Timeline  inherits  similar  improvements  that  are  made  for  the  Map  and  Timeline  GUI. 

The  following  themes  emerged  from  the  comments: 

s.  The  app  highlights  the  need  to  analyse  temporal  patterns,  something  that  is  not  done 
much,  and  offers  tools  for  doing  that, 

t.  Participants  would  like  to  use  this  with  trip-wire,  choke  point,  and  region  of  interest 
events. 


Table  6  Comments  and  Observations  About  the  Events  Timeline 

Events  Timeline 

Average  User  Ranking:  1.5 
Positive  Comments: 

•  Great  for  comparing  tracks  of  the  same  vessel  [C.2.1.3e,  C.2.3.1e,  C.2.4.3e] 

•  Temporal  representation  is  a  topic  we  need  to  become  accustomed  to  using  [2.2.1 .1g] 

•  Very  good  representation  [C.2.3.1e,  C. 1.2. 3b,  C.3.3.3a] 

•  Liked  multi-timelines  with  ability  to  unlock  and  shift  them  [C.3.3.3e] 

Evidence  of  Self-Learning: 

•  Figured  out  how  to  remove  a  split  timeline  [C.1.1.2j] 

•  Had  good  skill  with  the  timeline  [C.1 .4.2a] 

Suggested  New  Ways  to  Use  This  App: 

•  Synchronize  on  arrival  at  a  specific  port  then  look  at  previous  ports  of  call  [C.3.3.3f] 

•  Compare  tracks  of  the  same  vessel  over  multiple  voyages  [C.2.1.3e,  C.2.3.1e,  C.2.4.3e] 

•  Useful  for  investigating  close  encounters  and  fleet  activity  [C.2.4.3e] 

•  Feed  trip-wire  events  or  other  “alerts”  into  the  app  [C.3.2.3p,  C.3.1 .4p] 

Noted  Bugs  or  Criticisms: 

•  Not  very  intuitive  [C. 2.2.1. 1  g] 

•  Not  applicable  to  my  position  [C.2.3.3d] 

Observed  Errors  When  Learning: 

•  (none  reported) 

Suggested  Improvements  or  Added  Features: 
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•  Provide  a  legend  explaining  what  the  icons  mean  [C.2.1 .4e] 

•  Type  in  dates  rather  than  scroll  the  slider  [C.2.3.2c] 
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7  The  Route  Forward 


The  results  of  these  trials  should  be  influential  in  identifying  the  next  step  forward  to  make  these 
visual  analytics  tools  and  concepts  operationally  available  to  the  CF  and  the  MSOC.As  summarized 
in  Section  6,  comments  from  the  trial  participants  are  very  positive,  but  to  achieve  this  potential 
operational  migration,  foundations  must  first  be  laid  as  discussed  in  Section  7. 1 . 

7.1  Steps  required  Before  an  Eventual  Operational  Deployment 
at  the  MSOCs 

The  MSOC  user  community  is  keen  to  start  using  MVAP  soon,  as  evidenced  by  comments  such  as 
the  following  from  Table  1 : 

•  “Between  other  systems  or  MVAP,  this  is  the  best” 

•  “Very  useful  -  can’t  wait  to  use  it,  when  do  we  get  it?” 

•  “This  is  a  tool  that  an  analyst  would  use.” 

This  section  briefly  reviews  what  has  to  be  done  to  achieve  this. 

There  are  many  challenges  associated  with  raising  a  prototype  such  as  the  MVAP  to  operational 
status,  but  the  MVAP  architecture  was  chosen  to  make  the  transition  less  difficult.  DRDC  and 
Salience  decided  in  an  early  design  study  that  MVAP  would  be  a  set  of  apps  rather  than  an 
integrated  system  solution  [10].  It  was  clear  at  the  time  that  the  target  users  -  the  naval  RJOCs  and 
the  multi-agency  MSOCs  -  would  never  fund  the  development  of  a  new  integrated  system  solution 
just  to  gain  visual  analytics  technology.  MVAP  was  thus  envisioned  as  an  ensemble  of  loosely- 
integrated  apps  that  could  be  attached  onto  an  existing  system  as  value-added  modules. 

To  maintain  this  value-added  app  approach,  DRDC  and  Oculus  implemented  the  apps  using  a 
Service  Oriented  Architecture  with  code  written  in  JavaScript  and  Java,  and  libraries  such  as  Google 
Web  Toolkit,  jQuery,  and  Oculus’s  Aperture  [12],  This  means  that  the  prototype  can  be  run  on  a 
generic  browser  as  long  as  appropriate  web  services  are  in  place  to  serve  up  the  operational  datasets. 
The  technical  challenges  of  connecting  the  MVAP  widgets  to  MSOC’s  operational  data  are 
therefore  readily  achievable. 

The  following  tasks  will  be  required: 

•  Characterize  the  MSOC  IT  Context:  Acquire  or  create  a  detailed  specification  of  how  the 
MVAP  apps  will  connect  into  the  MSOC  systems.  MSOC  systems  were  characterized  at  the 
beginning  of  the  MVAP  development  [5,  7]  but  that  analysis  needs  to  be  updated  because 
major  changes  have  occurred  at  the  MSOCs  since  then.  Work  will  include: 

♦  Create  a  list  of  all  interfaces  that  can  and  should  be  used  by  the  MVAP  when  it  is 
installed  in  the  MSOC. 

♦  Collect  the  specifications  for  those  interfaces,  either  by  reading  existing  documents  or 
by  visiting  the  MSOC  and  creating  a  new  document. 
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♦  Identify  current  MSOC  analysis  tasks  that  would  be  affected  by  the  MVAP  apps,  and 
characterize  their  objectives,  data  quality,  work  flows,  etc.  Clarify  exactly  what 
products  each  MVAP  app  should  produce  in  order  for  those  products  to  integrate  easily. 

♦  Identify  data  firewalls  within  the  MSOCs  that  will  constrain  MVAP’s  connectivity. 

♦  Determine  what  accreditation  process  will  be  required  by  MSOC  IT  management 
before  they  will  allow  the  installation  of  apps. 

•  Specify  Essential  Modifications:  based  on  the  above  description  of  the  operational  context, 

plus  lessons  learned  in  these  trials,  specify  all  essential  MVAP  modifications.  These  are 

expected  to  include: 

♦  Support  for  new  interfaces  between  MVAP  and  the  MSOC  systems. 

♦  Ability  to  work  in  a  dynamic  information  space  where  contact  information  is  constantly 
being  added. 

♦  Modifications  to  MSOC  components  that  are  external  to  the  MVAP  apps  (e.g.  adding  a 
new  table  to  an  MSOC  database,  or  adding  a  new  service  to  an  MSOC  data  stream). 

•  Prioritize  and  Select  Optional  Improvements:  create  a  prioritized  list  of  non-essential  but 

desirable  improvements  to  the  MVAP  apps: 

♦  Based  on  the  November  2013  trial  results  and  on  information  collected  from  the 
MSOCs,  create  a  list  of  optional  MVAP  improvements  and  rank  them  by  cost  and 
benefit. 

•  Develop  an  integration  plan: 

♦  Create  formal  plans  for  implementing,  validating,  installing,  and  integrating  the  MVAP 
apps 

♦  Validate  the  plans  with  the  scientific  authority  and  with  the  MSOC  agencies. 

•  Implement  Selected  Software  Improvements: 

♦  Write  the  new  software. 

♦  Test  the  software  as  much  as  possible  in  a  simulated  environment. 

•  Install,  Validate,  Train,  and  Support 

♦  Install,  test,  and  validate  the  software  in  the  MSOC.  The  initial  connection  would  be 
one-way,  so  that  errors  made  in  the  MVAP  apps  do  not  trickle  down  to  the  MSOC. 

♦  Train  MSOC  operators  on  the  MVAP  (with  the  one-way  data  feed),  assess  their 
abilities,  assess  the  reliability  of  the  MVAP  elements,  and  determine  whether  MVAP  is 
ready  to  go  live. 

♦  “Go  live”  by  connecting  MVAP  output  products  into  the  MSOC  data  stream. 

♦  Provide  long-term  software  support. 
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8  Summary  and  Conclusion 


Phase  Three  of  the  Maritime  Visual  Analytics  Prototype  (MVAP)  contract  has  been  successfully 
completed.  This  document  is  the  Final  Report  for  Task  2,  Prototype  Apps  Validation 

The  objectives  of  the  validation  trials  were  to: 

•  Collect  objective  evidence  about  the  value  of  the  MVAP  apps  to  support  rapid  situation 
awareness,  visualization  of  the  recognized  maritime  picture,  anomaly  detection,  and 
collaboration. 

•  Identify  how  the  apps  could  be  improved  to  better  match  end-user  requirements. 

•  Take  an  initial  step  toward  installing  the  MVAP  in  an  operational  venue. 

Trials  Planning 

The  project  team  completed  the  following  tasks  in  preparation  for  the  trials: 

•  A  User  Jury  methodology  was  selected.  Groups  of  skilled  users  from  the  MSOCs  would  be 
briefly  trained  to  use  the  MVAP,  given  relevant  tasks  to  perform  in  a  hands-on  exercise,  and 
then  asked  to  assess  the  MVAP  apps. 

•  An  early  onsite  planning  session  was  held  in  Halifax  ten  months  before  the  validation  event. 
Nineteen  MSOC  analysts  representing  the  RCN,  Coast  Guard,  RCMP,  and  CORA  were 
briefed  on  MVAP  capabilities  and  gave  suggestions  that  influenced  the  tasks  and  datasets  for 
the  User  Jury. 

•  The  following  validation  metrics  were  selected  (see  Section  2.4):  observations,  anecdotes,  time 
on  task,  app  ranking,  and  System  Usability  Scale  (SUS). 

•  Trials  datasets  were  assembled  using  archived  Global  Positioning  Warehouse  (GPW)  data 
from  the  MSOCs.  In  one  case  it  took  tens  of  hours  to  sift  the  trial  data  from  a  very  large  GPW 
archive. 

•  Scripts  were  created  for  both  training  the  participants  (Section  3)  and  for  the  hands-on  exercise 
(Section  4). 

The  Use  Jury  trials  were  held  November  13th  and  14th  2013  in  a  computer  lab  located  in  the  same 
building  as  the  MSOC  watch  floor.  Participants  were  from  most  of  the  MSOC  agencies,  including 
the  Royal  Canadian  Navy,  RCMP,  Transport  Canada,  Canadian  Border  Services  Agency, 
Department  of  Fisheries  and  Oceans,  and  Coast  Guard.  Observers’  notes,  participants’  comments, 
and  rankings  were  collected  on  four  different  paper  forms,  analysed,  collated,  and  transcribed  into 
Tables  1  through  6  of  this  report.  The  original  hand-written  forms  are  reproduced  in  Annex  B.  The 
only  major  deviation  from  the  plan  was  that  the  observers  were  not  able  to  accurately  estimate  the 
time  for  each  task. 

Trials  Results 

As  an  overall  summary  of  the  success  of  the  MVAP  apps,  the  System  Usability  Scale  results  were  of 
particular  interest  because  they  could  be  compared  to  a  large  statistical  base  of  similar  surveys.  The 
average  SUS  score  was  76/100,  which  is  considered  “excellent.” 
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Verbal  reports  from  the  participants  were  even  more  enthusiastic.  As  participants  arrived  for  the 
third  session,  they  reported  that  there  was  a  “buzz  on  the  watch  floor  that  the  MVAP  is  pretty  cool.” 
Some  of  the  glowing  comments  (paraphrased)  included: 

•  “Between  other  systems  or  MVAP,  this  is  the  best” 

•  “Very  useful  -  can’t  wait  to  use  it,  when  do  we  get  it?” 

•  “This  is  a  tool  that  an  analyst  would  use.” 

•  “The  GUI  was  intuitive  enough  that  participants  were  trying  features  without  waiting  for 
training” 

•  “All  apps  represent  concepts  that  would  greatly  improve  the  visual  analysis” 

Many  of  the  comments  indicated  that  participants  were  thinking  ahead  to  how  they  could  bring  the 
full  advantages  of  the  MVAP  to  the  watch  floor.  They  pointed  out  for  example  that  it  will  be 
necessary  to  exchange  data  with  existing  MSOC  systems  such  as  Greenline.  They  postulated  that  a 
flexible  and  deeper  structure  could  be  put  in  place  so  that  the  Analysis  Sets  can  conform  to  a  wide 
variety  of  analytical  problems.  They  wondered  if  the  content  shown  on  the  Vessel  Summary  Cards 
could  be  user-configurable.  They  brainstormed  about  bringing  new  types  of  calculated  values  into 
the  Magnets  Grid.  They  asked  whether  MVAP  could  also  be  used  in  a  context  disconnected  from 
the  MSOC  data  grid. 

The  Map  and  Timeline  app  was  flagged  by  some  as  “the  best  app  of  all.”  Participants  appreciated 
how  it  visualized  time  evolution,  showed  vessels  interacting,  and  displayed  Close  Encounters 
information.  They  stated  that  it  is  almost  impossible  to  detect  close  encounters  with  the  tool  suites 
currently  at  the  MSOCs. 

The  Magnets  Grid  app  also  drew  a  lot  of  praise  (it  was  called  an  “awesome  representation”  for 
example)  because  it  offers  a  completely  new  way  to  explore  numerical  and  textual  attributes  of 
vessels.  Participants  particularly  admired  how  they  can  drag  and  select  a  subset  of  the  dots  to  export 
as  an  analysis  set. 

The  trials  also  revealed  ways  that  the  MVAP  apps  might  be  improved,  and  these  are  also  listed  in 
Tables  1  through  6.  The  best  example  of  this  is  the  need  to  improve  the  Time  Slider  interface  used 
in  both  the  Map  and  Timeline  and  the  Events  Timeline  apps,  as  discussed  at  length  in  Section  6.5. 

Follow-On  Planning 

Section  7  discusses  options  for  building  on  the  success  of  the  trials  and  moving  toward  installing  the 
apps  operationally.  It  recommends: 

•  A  brief  initial  study  to  characterize  in  detail  how  MVAP  apps  must  integrate  with  MSOC 
systems. 

•  A  prioritized  list  of  enhancements  to  the  MVAP  that  should  or  could  be  done  in  preparation  for 
deployment. 

•  A  staged  deployment  that  includes  validation,  training,  and  long-term  support. 
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8.1  Conclusions 


Phase  3  validation  of  the  Maritime  Visual  Analytics  Prototype  apps  has  successfully  achieved  the 
contract  objectives.  User  Jury  trials  conducted  with  analysts  from  six  MSOC  agencies  have 
confirmed  that  MVAP  apps: 

•  Support  rapid  awareness  and  identification  of  key  information  elements  and  thus  rapid  insight 
into  the  meaning  of  a  situation. 

•  Improve  the  visualization  of  the  recognized  maritime  picture. 

•  Enable  better  comprehension  of  a  situation  and  how  it  could  develop. 

•  Support  detection,  alerting,  and  visualization  of  anomalies. 

The  resounding  conclusion  from  these  trials  is  that  the  MVAP  apps  should  be  integrated  into  the 
MSOCs  as  soon  as  possible.  In  support  of  that  goal,  this  document  has: 

•  Established  rigorous  evidence  that  can  be  used  to  promote  the  MVAP  approach  to  operational 
users,  and 

•  Provided  a  description  of  the  steps  that  need  to  take  place  for  eventually  deploying  MVAP  to 
the  MSOC, 
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Annex  A  Worksheets  Used  for  the  Trials 


The  following  pages  show  the  worksheets  that  were  handed  out  to  the  participants  and  the 
observers  for  the  scenario-based  assessment  tasks,  as  described  in  Section  4. 1 : 

•  Observer  Worksheet:  Observers  used  this  sheet  to  record  their  observations  during  training 
and  tasked  activities.  Each  observer  was  assigned  to  one  participant,  and  thus  each  sheet 
describes  observations  of  one  participant.  The  completed  worksheets  are  reproduced  in 
Annex  B.l. 

•  Scenario-Based  Worksheet:  Participants  used  this  sheet  to  record  their  analysis  insights  for 
the  six  tasks  as  they  progressed  through  the  exercise.  These  worksheets  served  primarily  to 
help  the  participants  move  in  a  structured  way  from  task  to  task.  This  information  was  not 
analysed. 

•  System  Usability  Survey:  Participants  ranked  the  MVAP  using  a  System  Usability  Survey 
(SUS)  questionnaire,  as  discussed  in  Section  2.5.  The  filled-out  SUS  sheets  are  reproduced 
in  Annex  B.4. 

•  Participant  Rankings:  After  completing  the  exercise,  participants  ranked  the  apps  and 
provided  further  comments.  The  completed  worksheets  are  reproduced  in  Annex  0. 

•  Hot  Wash-Up  Notes:  Observers  kep  notes  of  comments  made  during  the  Hot  Wash-Up.  The 
completed  sheets  of  notes  are  reproduced  in  Annex  B.3. 
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Observer  Worksheet 


Participant:  _  Observer: 

Questions  Asked  by  the  Participant: 


Non-Critical  Errors  During  Training: 


Non-Critical  Errors  During  the  Tasked  Exercise: 


Evidence  of  Self-Learning  (Strategies  Improving  with  Time): 


Critical  Errors  (Participant  had  to  get  help  to  continue): 


(Exit  Interview  is  on  the  other  side) 


System  Usability  Survey 


Mark  one  box  in  each  row. 


1 .  I  think  that  I  would  like  to  use  these  apps 
frequently 


2.  I  found  the  apps  unnecessarily  complex 


3.  I  thought  the  apps  were  easy  to  use 


4.  1  think  that  I  would  need  the  support  of  a 
technical  person  to  be  able  to  use  these  apps 


5.  1  found  the  various  functions  in  these  apps 
were  well  integrated 


6.  I  thought  there  was  too  much  inconsistency 
in  these  apps 


7.  I  would  imagine  that  most  people  would 
learn  to  use  these  apps  very  quickly 


8.  I  found  the  apps  very  cumbersome  to  use 


9.  I  felt  very  confident  using  the  apps 


10. 1  needed  to  learn  a  lot  of  things  before  I 
could  get  going  with  these  apps 


Strongly  Strongly 

disagree  agree 


1 

2 

3 

4 

5 

1 

2 

3 

4 

5 

1 

2 

3 

4 

5 

1 

2 

3 

4 

5 

1 

2 

3 

4 

5 

1 

2 

3 

4 

5 

1 

2 

3 

4 

5 

1 

2 

3 

4 

5 

1 

2 

3 

4 

5 

1 

2 

3 

4 

5 
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Participant  Ranking 

How  valuable  would  each  App  be,  if  integrated  into  the  domain  awareness  systems  that  you 
use: 


App  Name 


Analysis  Set 
Manager 


Rank  0  -  2 

0:  not  useful 
1:  possibly  useful 
2:  very  useful 


Record  Browser 


p - 1 

:: 

— 

[HI 

'■tot 1  t 

-- — -saSSs^r*** 

Magnets  Grid 

n—— , 

'•V, 


 ^  4 


Map  and  Timeline 

/'a 


Timeline 


Comments 


Other  Comments  (will  be  noted  by  the  observers): 
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Exit  Interview  -  Group  Hotwash 

What  are  your  general  impressions  of  how  this  type  of  functionality  might  impact  your  work? 


What  were  the  most  useful  features/strengths  of  the  applications? 


What  were  the  biggest  roadblocks  in  completing  the  Carnival  scenario? 


Do  you  have  any  thoughts  for  how  the  apps  could  be  improved? 


Any  other  thoughts? 


Annex  B  Raw  Notes  and  Filled-Out  Worksheets 


Annexes  B.l  through  B.4  reproduce  the  original  hand-written  observations  and  participant 
responses  from  the  trials  conducted  in  Halifax  in  November  2013.  These  are  cross-referenced  from 
the  discussion  in  Section  6  in  case  there  is  some  question  about  exactly  what  was  meant  by  a 
comment. 


B. l  Observations  of  the  Exercise 

This  Annex  shows  the  hand-written  notes  that  the  observers  made  during  the  training  and  the 
exercise.  These  are  summarized  in  Section  6,  so  each  note  is  marked  with  a  code  that  can  be  cross- 
referenced  from  there.  Table  7  is  an  index  into  the  Observer  Notes  pages.  The  total  number  of 
observation  sheets  differs  from  the  number  of  participants  the  observation  sheets  C.  1.3.1  and 

C. l.3.2  are  about  the  same  participant. 


Table  7  Observer  Worksheets 


Number 

Observer 

Time 

C. 1.1.1 

Valerie 

C. 1.1.2 

Vincent 

Nov  13  15:30 

C.1.1.3 

Bill 

C. 1.1.4 

Mike 

C. 1.2.1 

Vincent 

C. 1.2.2 

Valerie 

C.1.2.3 

Bill 

Nov  14  10:30 

C. 1.2.4 

Valerie 

C.1.2.5 

Mike 

C.1.3.1 

Vincent 

C.1.3.2 

Valerie  and  Vincent 

Nov  14  13:00 

C.1.3.3 

Bill 

C.1.3.4 

Mike 

C. 1.4.1 

Bill 

C. 1.4.2 

Vincent 

Nov  14  14:30 

C.1.4.3 

Valerie 

C. 1.4.4 

Mike 
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Original  Observer  Worksheets 


Eculus 


\3  ^ 


IMh  ‘"7 

|  lidk, 

■I 


Participant: 


Observer  Worksheet 

— - - Observer:  l/dlitif. 


Questions  Asked  by  the  Participant: 


C. 1.1. la 


C. 1.1. 1b 


Mon-Critical  Errors  During  Training^ 
fl5f 1  'Pifck  <iUrxli  hit  ik! 

ftkr  WnS,  A-.'iJ  Ufti'htj  ■fk  //»■/*■</  L\tft  Se(i ct  <//  ^ 

/V| ‘7>u!^Am'  ^  ^  XOflMJ  JiUftJ  dt.li  ,’j  *,/",/  y,’xtAr 

fU  ftW  '*k  ft  0**4  /V  a*/‘a.iK  **•£«,  j\  ,hjW»/) 

/cS  l/SC  fr;t^  ft  £i#*  <W  A«i/i 

a t-f-M*ck4 


C.l.l.le 

Mon-Critical  Errors  During  the  Tasked  Exercise: 

/?5A  clftk  AwWa/  fO'rA 


C. 1.1. Id 


C.l.l.le 


C.l.l.lf 


C.l.l.lg 


Evidence  of  Self-Learning  (Strategies  improving  with  Time) 
«flrfh  *4  Lk4.4  »/  7WuV<>*  (Vi  My  1 

li,tU  <lP  Wu*/ 


C.l.l.lh 


Critical  Errors  (Participant  had  to  get  help  to  continue): 

hot  t-j 


C.l.l.li 


C.l.1.1 


Task 

Tim©  (min) 

Start 

* 

1 

3h.v> 

2 

3 

At  J 

4 

At 

5 

1  m-h 

6 

1  M.% 

Eculus 


4  »» 


0  v/ 


R|>- 


C. 1.1. 2a 


/f 


Observer  Worksheet 

Particip..,:  _  Ob*tvcr: 

Questions  Asked  by  the  Participant: 

'JjV>V^  ^°Ua  CMTv  1m  ^  '’Ci/ib  ^C(0  ^  4  ^ _ | 

,***  ^Kui  lA  u'w'  Aj-^r 

ion-Critical  Errors  During  Training: 

'  —i  Uja.  ivM  J  ^ 

-  &  A  |  c. i.i. 2d  | 


2b 


C. 1.1.2c 


£*44 


rHL  /^Vf. 


Vj  pNon-Critical  Errors  During  the  Tasked  Exercise: 

^  k  uij  J 


C.1.1.2e 


C.1.1.2f 


^J>s  *.  UW  ^„..p  ^ 

"  /1A  t„rj  £ 

*  SUk^.  |  a  /  .  , 

^  *j.r  0^, 

Evidence  or  Self-Learning  (Strategies  Improving  with 

'  #M- _ c.  sj,,[,fu  t 


C.1.1.2g 


'(K*aA> 


C.1.1.2h 


C .  1 . 1 . 2  j 


C.1.1.2i 

t  line): 

V./ 


Critical  Errors  (Participant  had  to  get  help  to  continue): 


C.l.1.2 


Task 


6 


Time  (min) 


2  %(x. 


L  ^ 


{at  *  n 

(  J  'Try  \*~l 


^  -y~n.S) 


■  ■’m 


.  \ 

,k*>f 


Eculus 


& 


'If 


Aj%00<Sf^K. 

J* 


A 


no7- 


'“►*-<«  I  <J  J  Mnmi 


\  W°/ 


Participant: 


Observer  Worksheet 

________  Observer: 


B 


i  V'L- 


Questions  Asked  by  the  Participant: 

t/K  Wo  ••W.'Vi 

i  _**.  .  w  *4-  '  t 


- 

cUo  ^4-  fijJlfaJu 

*'••■  '  j/tiA&y  ov  tsL  A**. 


fna*  J  ^  J 


j  i  C  1.1.3a 
V--i  - - - 


C. 1.1. 3b 


Non-Critical  Errors  During  Training: 


C. 1.1. 3a 


Non-Critical  Errors  During  the  Tasked  Exercise: 


Evidence  of  Self-Learning  (Strategies  improving  with  Time): 

haiAa^  Ca^M  ivOCii.%4 


C. 1.1. 3c 


C. ritical  Errors  (Participant  had  to  get  help  to  continue): 


C. 1.1. 3d  ]o>i 


C.1.1.3e 


v4*- . *J°  A  •&-!,  '2-  '^WytV' 


C.1.1.3f 


kf  GW,  {dhK 


Task 

Time  (min) 

Start 

1 

2 

3 

4 

5 

6 

C.1.1.3g 


^  Ui~  tT 


4» . J) . . |u^ 


C.  1.1.3 


Participant: 


Observer  Worksheet 

Observer: 


C. 1.1.4a 


Questions  Asked  by  the  Participant:  no  M 

_  -v.  fp»J*+***  y~* 

<r 


C. 1.1.4b 


)&£?'**■  '  **’— po-*~ 


•4«J[  6\U^~  o'" 


Non-Crilical  Errors  During  Training:  . 

r*"V>^  e^Vw>  -  1  U<_  ^  . 

Urv.  4»  ^  4ta  4wUx  -  ^.cl  *»^«  ***  p 

— 114J,  Thoja  iL^.i^y.,1  -?j^r~-~r= 


A 

%et j  C. 1.1.4c 


Q 

/ 


*  V4*^'  <p"  *  *WW*|  C.1.1.4f 

Nan-Critical  Errors  During  the  Tasked  Exercise: 

-  G^*"***!^  w^JT1  f-  %taau<k.Vc<.v  joer-jp-c  S<Cw*-^ 

-  OiJ^h'  r  rfx&*s-  tt^<_  G*P  kJ  ■tn^.'t 


C.1.1.4e 


C.1.1.4g 


C.1.1  4h 


C.1.1.4i 


Evidence  of  Self-Learning  (Strategies  Improving  with  Time): 

..  i'-^W,  '<>*(*''•<.  £r'»C  • 

6Jtr  c\  £**><-  o>v  GwjXt,0'.«_^.  s«Uw-jL. 

-  Inr»e*l  s  "Vi*-*  kj>1»jAu 

*T  r-Se*L  4e»  V  Cjo-^  r>.  w.%  » tfH-e.*-  -fo^Ls 


C.1.1 ,4j 


C.1.1. 4k 


Critical  Errors  (Participant  had  to  get  help  to  continue): 

St*VAaWr-~i  apt  pfc.-\^vwi»  «.(k-»~ 

iV>  «^xj.  4C*  eLf-nut  <»^vJL  cJLf«Y) 

'/tjr  m.  \^-<s.  rxpjpjU  usexjLl 


C.1.1. 4m 


6wt- 

5* 

0retv-p4-  -K.V^  4,  -((ip 
fC-c  tau-vC.  de-tr-  ~h»  f 

^e.j-  i-u^c  »( 


Task 

Time  (min) 

Start 

1 

4-*.  5e> 

2 

5:  /  o 

3 

^  :51b 

4 

<93*  /o.'M- 

1  5 

4T1'  |«fr-  j  j  ,^-j 

6 

■tGZSST  12:45' 

C.1.1. 4 


Ecuius 


pu  \<\  |o3 


asrswca  I  J  Mnm 


Observer  Worksheet 

f 

P,,1kil“n,;  _ _  Observer: 

Questions  Asked  by  the  Participant: 


'-Au.We  i  **ftt*T~  Y  (h*h*<Jj 


C. 1.2. la 


C. 1.2.1b 


Non-Critical  Errors  During  Training: 

-  sJi  -J  t-*»J  ( ' t, 


J*'  X*  «lu 


£»  t '  ^IU 


-  ££**0/  uu/1^ 

Non-Critical  Errors  During  the  Tasked  Exercise: 


^  3'c°>~'  r*h*  {,wsd L. 


Evidence  of  Self-Learning  (Strategies  improving  with  Time): 


Critical  Errors  (Participant  had  to  get  help  to  continue): 


C. 1.2.1 


Task 

Time  (min) 

Start 

1 

2 

‘Z*  #**„,« 

3 

3» 

*m~  pj 

4 

L  -JMn 

5 

«  * 

» 

6 

\  b 

1 

Observer  Worksheet 

Participant:  _  _  Observer: 

Questions  Asked  by  the  Participant: 

A 

Non-Critical  Errors  During  Training: 
fjjj  ft (Jl 


Non-Critical  Errors  During  the  Tasked  Exercise: 

(U*/ 


Evidence  of  Self-Learning  (Strategies  Improving  with  Time): 

14  «s  <$A«lsh  /v,  rU  .’W/V^ 


Critical  Errors  (Participant  had  to  get  help  to  continue): 


C. 1.2.2 


Test 

( 

Time  (min) 

Star 

t 

tlhM 

1 

ays 

2 

itusHU'h 

3 

4 

//j.  >o  *  f /l  *1 

5 

6 

W>J-  IU1 

BSculus 


Participant: 


Observer  Worksheet 

-  _  Observer: 


-{V  It 


I  o?o 


ft  <  w' — 


C.1.2.3a 


Questions  Asked  by  the  Participant: 

V>  'ftt>  ,3-v.  A*-f  ^ .  C  ^  V  jf 


C. 1.2. 3b 


^•CSWYH 


i  1l  I  I  .  / 

,S7  /)  r  V<Sk/«<>] 


C.1.2.3c 


Non-Critkal  Errors  During  Training: 


.Non-Critical  Errors  During  the  Tasked  Exercise: 

wr  v  yua^u 

&*-i fM  «4t»  $cJL*4fr~  0 

/  j.  *  t  3  ,iri|l 


C. 1.2. 3d 


C.1.2.3e  1  Wn^  2 . 

2^,  _  _ 

JLu>r  ^  Wukjs-^  ^  0*v«£l^ 


M .  [yv'^W  | 

C**Ml  (rJ  ^ . .  .  •***  'r+-njj 


_  .<?yv»  / 
ebuix  cU"ik^. 


C.1.2.3f 


’  <yp 


Evidence  of  Self-Learning  (Strategies  Improving  with  Time): 


C.1.2.3g 


^r\ 


Critical  Errors  (Participant  had  to  get  help  to  continue): 

‘rt  ^  ^  -Ck^ 

J^»  iNt£r4*  ^  (f)Vr\  “fo 

Ai  <5 

C.1.2.3h  ^*Vi^  ul  Ifl#  CaX.  Pi)  •' 'h 

J  ‘  AX  *  *  *  V 


Ta.sk 

Tims  (min) 

Cn^V 

Start 

to 

■ 

2 

3 

4 

5 

( 

5 

C.1.2.3i 


  w  c{ 


f  <Jr  ifc1  Aj 

5  ^  4*  p*  / 1  i)<^x  Aajw 


C.l.2.3 


t^vwe.  <rf 

r  TR  f*nts»r  numkar  knlf  DOOO  fUWI 


<N0L 


Eculus 


(Ucm/  H 


1° 


3* 


OKflMU 


Participant: 


Observer  Worksheet 

_ _  Observer: 


i C/m 


C. 1.2. 4a 


C. 1.2.4b 


C. 1.2. 4c 


iiestions i  Asked  by  the  Participant: 

Codl  I  /ptt|4  -cite  **  /tMi/i Ac  A  i  sptoifre  Jjua  ? 

I^C  Cm./#/  J  }^1  /tQ-f  7 

ddv  Qfy  fh*  Ac#//  wfnUrt  I**  H  9*"'  A**I‘  Ly  #*./(« 


C.1.2.4d 


^Non-Critical  Errors  During  Training: 


C.1.2.4e 


C.1.2.4f 


C.1.2.4g 


ftyi  Y**n§  (§&  A^//A/<t ti 

fap  rmJ<J  hit  ivlh 

y/W"!)  ht*,r  teJ-  cv\tj  tuf  AIcI 

4$*  tynuJ,  KE  ***■  sA.^ 


Non-Critical  Errors  During  the  Tasked  Exercise: 


C.1.2.4h 


C.1.2.4i 


M  scUJ^ (j  ^ 

UU^eliM  Aw-A?6  i,/  u,t  /U  Uut^tU- 

id  mt-  wr  m  //*/*« 

C**+£A 


Evidence  of  Self-Learning  (Strategies  Improving  with  Time): 

C.1.2.4j  ##*  r****toU  jj i/Utuff 


C  ritical  Errors  (Participant  bad  to  get  help  to  continue); 


C. 1.2.4 


Task 

Time  (min) 

Start 

(»*t/Y 

1 

1  Uf 

2 

tiur-wl 

3 

ikn-m 

4 

5 

6 

ntf  -t  a  9S~ 

Kculus 


\H 


^3  * 


no- 


Jm>uu 


Observer  Worksheet 

_  Observer:  1^*4  jtp  _ 

yuraiuns  -vskcu  Dy  me  rurtjcipapt: 

AsWjl  V  cl  r\  -\La. 

<3onK  ^°P  4Stvv-f*  44'f *-(  c  <*-*<?(  aJ~  -So-^-c.4-  . 


Participant: 

Questions  Asked  by  the  Participant: 


Qf*c  A***  top  <su^A. 

(P<-oUW»v-v  u)A4«.  -fc-jci- 


C. 1.2.5a 


1.2.5b  | 

K*e  4rt»«i  4»  cUx*^ju.W-:  ^  *t.  v-e  wej  ,  —  <m4  >a  £~«w  *\\ 


C. 1.2. 5c 


<rw.c-c_ . 


^roSi-pAro^Y  t»>  tlAJSVMU.'H 

Lg>ic,>s^  *4  l;«<  aliitt.  i j, 

NwCriticiMirpoHi  Durtnf  Trainlug:  4  — - - 

u)<wWv  4t-  sdk^k^  ^U.  4^c-  C*-rnv/-JI  reswli  j.  «j(- 
T  ,  fl  4- 

Ir^eA  /M^iVe-  <5v«-^  4U  -\  lau^  -£>  -^u<  ce«^JrJ 


Non-Critical  Errors  During  the  Tasked  Exercise: 

f^J0< 


C.1.2.5d 


C.1.2.5e 


Evidence  of  Seif-Learning  (Strategies  Improving  with  Ti 
Xo-fc4L4<*  S  <V^J2X  -Va<uL  b*w<,  \W  "{{***.  It  iv«v- 
-f\^*v«jL  D-kA  SauiXz-t^ 


C.1.2.5f 

irVteiiHX 


C.1.2.5g 


1  <-:e^  t*-4  i^iOAS  *y? 
^XpU^Aja-MPlA*  ^ 


0  ij  VA.  o-wjl  «>cpa»'A  "Vo 

(0o*S.4irKS 
W  rvve^y 


C.1.2.5h 


Critical  Errors  (Participant  had  to  gel  help  to  continue): 

NJcpkJJ  • 


C.  1.2.5 


Task 

Time  (min) 

Start 

O',  oo 

1 

l‘Cf f 

2 

3'JZ 

3 

t.’sz  Ti, 

4 

6  3?>  ^  £ 

5 

g  .4*1  ^ 

6 

4-4 


33 


l  'DC  > 


Ecu  I  US 


A/W 

yJl jj± 


°««w  |  !•  1  Mm  ml 


Observer  Worksheet 


Participant: 

Questions  Asked  by  the  Participant: 


Observer: 


^rv^/ 


Nou-Critical  Errors  During  Training: 

-  H 


C. 1.3. 1b 


C. 1.3. 1c 


Non-Critical  Errors  During  the  Tasked  Exercise: 


Evidence  of  Self-Learning  {Strategies  Improving  with  Time): 


Critical  Errors  {ParticSpant  had  to  get  help  to  continue): 


C. 1.3.1 


K  f h  ^  Isvtpvrt  ■OOAj  ylU+Juf  1 


C. 1.3.1a 


Task 

Time  (min) 

Sta 

rt 

1 

2 

1  rrt.Vv 

3 

2 

j  m  i  »v. 

4 

2-w 

5 

^  Or\  |  a 

6 

1  VtlAJ 

sculus 


/Ogv  )i  / 


Non-Critical  Errors  During  Training: 
p\dM  ok  M?W  tt%e*nh  / 

fry  emir*!  ifUicuff  «/  /h/ 


Participant: 


Observer  Worksheet 

— _ ___  Observer:  jtlirit.  -f  \J iVirt  f~ 


Questions  Asked  by  the  Participant: 


r, 

1  ^  ?a 

C. 1.3. 2a 

Non-Critical  Errors  During  the  Tasked  Exercise: 

Went  Jw,// 


Evidence  of  Self-Learning  (Strategies  Improving  with  Time): 

Ufa  sk^  trn»  #6  fhdirrcUl  c^d | 

f h*M  of  #. 


C. 1.3. 2c 


Critical  Errors  (Participant  had  to  get  help  to  continue): 


C.l.3.2 


Task 

Time  (min) 

Start 

1 

/ 

2 

/w-JK. 

3 

•5  i»4\ 

4 

l  *WV\ 

5 

6 

1  V*T>-\ 

Eculus 


Participant:  __ 
Questions  Asked 


)  j>~> . . . j. 


nmv+mcm  J  J  M*wh 


Observer  Worksheet 

_ _ _ _  Observer:  jT . ' 


I U . \ 


by  the  Participant: 


Non-Critical  Errors  During  Training: 


N'on-Critical  Errors  During  the  Tasked  Exercise: 


C. 1.3. 3a 


Evidence  of  Self-Learning  (Strategies  Improving  with  Time): 

'“I'M  .  ftL^Ck 

vmIo  k  uJr 
iyJL.  vjc^A“  l&d*-'  <t . 

Critical  Errors  (Participant  had  to  get  help  to  continue): 

llcbJ  dv  V^K. .  .  kLzMkj 

— 1 4^  ^  Caa isi\V<^[  sixp; / 

,,,J^  fc*A  | 1  lx . |my\ 


C. 1.3.3b 


C.l.3.3 


Tas 

k 

Time  (min) 

Stai 

rt 

1 

2 

3 

4 

5 

6 

Eculus 


fOo \j  /y  j 


t 


nn 


WM9  RJ  j  »•»»« 

t. 


C. 1.3. 4a 


Participant 
Questions  Asked  by  the  Participant: 

rt/A  P  .(.'j  Iw-/  4  Vc«.  -H-tlj  ' 

-v  A-A..IS  4«* 

^  *-3. 


Observer  Worksheet 

_ _  Observer: 


C. 1.3. 4b 


C. 1.3. 4c 


fjxirtCV* 


•SCNtOj 

*=*<  Kr< 


C.1 ,3.4d 


Wk^t  14-  4*  *  ^0-^l  ^  id! -  ,1" 

Non-Critical  Errors  During  Training:  ^  *J^re  - _j 

U>Ue-<^  *"\  <4lMauiiKi  »-vr»vS  *|e»  wOt  eJL.  f  %**  ~f€et  ^  S~i^~  *'*'  *c-rv.i 

3 1  ^t-  vv"‘ca  ^  *\. 4o  ajfjL  */{*<.  yrc*'}+tJL-5 

cc-td  4®  A>^JL^,s  vi" 


C.1.3.4f 


C.1.3.4e 


Non-Critical  Errors  During  the  Tasked  Exercise: 

C.1.3.4g  "ca-t  *  *v©4  "  Cm.  *  O^- 

Cre.oli'^  *S 


C.1.3.4h 


Evidence  of  Self-Learning  (Strategies  Improving  with  Time): 

”T,S ^  ^  ^  4t.A^-ip.\  ^e3 

sW%c^U4  4»  i  «|; 


lf*W 


Critical  Errors  (Participant  had  to  get  help  to  continue): 


C.l.3.4 


Task 

Tima  (min) 

Start 

1 

5^s. 

2 

&  3&> 

3 

l  :  Id 

4 

2L 

5 

6 

i  ;2o 

Eculus 


(sf  8  v/  H 


P  st  JMnm 


a 


<V* 


•RO" 


Observer  Worksheet 

Participant:  _  _  Observer: 

Questions  Asked  by  tbe  Participant: 


i/l/vCAt- . ^ 


Noil-Critical  Errors  During  Training: 


Non-Critical  Errors  During  the  Tasked  Exercise: 


Evidence  of  Self-Learning  (Strategies  Improving  with  Time): 

VAy  lu,  }k  tun'd* 


C. 1.4. 2a 


Critical  Errors  (Participant  had  to  get  help  to  continue): 


C.  1.4.2 


Task 

Time  (min) 

Start 

1 

2 

3 

■"T: .  . . — - 

4 

5 

6 

Bculus 


{Jq>J  I  H 

TJ>°  ? . Vx 


Ri> 


Observer  Worksheet 

Participant;  _  _  Observer: 

Questions  Asked  by  the  Participant: 


C.l.4.3 


Non-Critical  Errors  During  Training: 

Ski#  elide  h  *<4/  Hi  A 


C. 1.4. 3a 


Click  M/flf  S b^fon  ^<Jud  AftUuM  b^Udk  A*} 


C. 1.4.3b 


Non-Critical  Errors  During  the  Tasked  Exercise: 


Bculus 


r*jy~ 


“•**•««  i  (i  )»>«» 

fc“* 


Participant: 


Observer  Worksheet 

Observer:  Mik 


Questions  Asked  by  the  Participant: 

^  <»>.p r>^  /\  C(t4c 


C. 1.4.4a 


Non-Critical  Errors  During  Training: 
hhejU/  -e»  f-crH-f  c.#*  r*\  4^^.  ti*< 

SkH  4mJola*  .  . 


C. 1.4.4b 


Non-Critical  Errors  During  the  Tasked  Exercise: 


Evidence  of  Self- Learning  (Strategies  Improving  with  Time): 


Critical  Eirors  (Participant  had  to  get  help  to  continue): 


C.  1.4.4 


Task 

Time  (min) 

Start 

t 

2 

3 

4 

5 

6 

B.2  Participant  Comments 


This  Annex  shows  the  hand-written  comments  written  by  the  participants  on  the  Participant 
Ranking  sheets  immediately  after  the  hands-on  exercise.  These  are  summarized  in  Section  6,  so 
each  comment  is  marked  with  a  code  so  that  it  can  be  cross-referenced  from  there.  Table  8  is  an 
index  into  the  Participant  Comment  pages.  One  participant  did  not  fill  the  ranking  sheet  in  the  last 
session. 


Table  8  Participant  Comments 


Number 

Association 

Role 

Time 

C.2.1.1 

RCN 

Coordinator 

C.2.1.2 

DFO 

Analyst 

Nov  13  15:30 

C.2.1.3 

RCN 

Lt(N) 

C.2.1.4 

RCN 

Intel  Officer 

C.2.2.1 

RCN 

Watch  Officer 

C.2.2.2 

CCG 

Analyst 

C.2.2.3 

TC 

Analyst 

Nov  14  10:30 

C.2.2.4 

RCN 

Lt(N) 

C.2.2.5 

TC 

Analyst 

C.2.3.1 

CBSA 

Intel  Officer 

C.2.3.2 

RCMP 

Analyst 

Nov  14  13:00 

C.2.3.3 

DND 

RMP  Manager 

C.2.4.1 

C.2.4.2 

GPW  developer 

Nov  14  14:30 

C.2.4.3 

47 


Original  Participant  Comments 


C.2.1.1 


C.2.1.2 


very  userui 

Analysis  Set 
Manager 

z 

Zc  cUaJ/'/'i  •Jo 

bqdctp  cosjafy s/d  Xd  J 

dOidjud  d->r  Pctcd'iCfJ 

Record 

Browser 

—  1 

rrb*sy  hof  dc  , 

xcc  cn*l  1  u<>uc</  —  o-Jzr  c<yiy>£>( 
crL  b€.  os£i4y 

d 

Magnets  Grid 

n —  - 

Sr; 

■  •  «..._ 

z 

(/\d  Cccd 

a.cce-^>  to  cUjtd 

Map  and  Timeline 

2_ 

'fa  i?t«y  -tCb.cI / y  Cj  ZcZ<~'  Q^jtr 
ActUx^  ,  0*5»-0 

Timeline 

'  ...  'll1. .  lb  % 

„  ooc2fo4w 

.  m,  oaoftorpp 

X 

tic/- &-■£>  //fZZ/o-f  hoyj-Cxs€r — 

C*r\  uo>-£  ryjoXf 

C^c)  ■j'O 

1 

Analysis  Set 
Manager 

'L 

v  ^‘vv^Izol  vs  \J<LVy  VV'C8_  . 

Record  Browser 

ill  a 

|  --3S£l-  !  -M  - 

I 

(j^Vy—Vw  t^oA^v.  4^  J  oJCfi.  . 

d 

/lagnets  Grid 

n— —  r» _ 7Z 

m  n*— 

•V*  Sr_- 

■  •*,  » — 

1 

\ 

^  ^  fv2  <»-e- Lt cf  <r-^v </  JLsy^&X 

"t®  Z'Z'/S-v/t  -dj/. 

Map  and  Timeline 

z 

UsV  r~*-  o  x/^x  .fJL . 

Timeline 

liiJil  -.1-  B 

•  *  m* 

rn-  OCCJ|Mn 

.  ... 

l 

riw<t - -  t/l  CoO  <t|  "tw^_  -4/otZs 

Other  Comments  (will  be  noted  by  the  observers): 

fJ 


.'l/OO  ' 


C.2.1 ,2f 


C. 2. 1.1a 


C.2.1. 1b 


C.2.1. 1c 


C.2.1. Id 


C.2.1. 1e 


C.2.1. If 


C.2.1. 1g 


C.2.1. 2a 


C.2.1 ,2b 


C.2.1. 2c 


C.2.1. 2d 


C.2.1.2e 


Analysis  Set 
Manager 

1  ***  ***'*** 

J3L 

Grtt^-  for  «jrc>v>ffrs^  Ce.rV'A-to 

Record  Browser 

WM  PM 

\ 

OSC^vA  t  l>\/f  A  C«>^lxrSa(r-e.  . 

Maanets  Grid 

O-x.  'tlmSZ 
n —  - 

% 

•  n*~'* 

—  ■ # 

^  •  «  «*— 

\ 

rOtwV  (  Uo4-  AoV- 

Map  and  Timeline 

0- 

Crr-t»V  t 

Timeline _ 

.  ...  «W” 

ccosgoefoo 

0- 

(Srtrvl"  f>r 

5jhv^  \ZtSs«A. 

C. 2. 1.3a 


C.2.1 .3b 


C.2.1 ,3c 


C.2.1 ,3d 


C.2.1.3e 


Analysis  Set 
Manager 

Record  Browser 

SkBag] 

1 S€- 

Magnets  Grid 

n—  - 

*•5* 

•^r.  Sr: 

.  f — 

1- 

Map  and  Timeline 

u 

Timeline 

1 

|  £..  vciy  uaeiui 

Analysis  Set 
Manager 

Record  Browser 

liifei 

ms.  -M- 

X 

Maanets  Grid 

n—  ■. 

A-.— 

»  W  i  2. — 

*  1  * 

.  ■*. 

Map  and  Timeline 

1 

fan  jiov  on  trft  plCff^y, 

HSOjtd  ftVwViP  /fo/i?  ftsfi?  to 

ProfP/  iy  Qva(oo\t*.  (goes  fc*~  ql(  gppij 

Timeline 

- 

"“ap*" 

1 

C.2.2.5a 


C.2.3.1a 


C.2.3.1b 


C.2.3.1c 


C.2.3.1d 


C.2.3.1e 


Analysis  Set 
Manager 


Record  Browser 


■■ 

^£5: 

3f 

Magnets  Grid 


C.2.3.2 


V 


Map  and  Timeline 


Timeline 


. 

OCCJpnlw 

. 

1:  possiDiy  usetui 
2:  very  useful 
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B.3  Hot  Wash-Up  Comments 


This  Annex  shows  the  hand-written  notes  made  by  the  observers  during  the  Hot  Wash-Up.  These 
are  summarized  in  Section  6,  so  each  comment  is  marked  with  a  code  so  that  it  can  be  cross- 
referenced  from  there.  When  a  Hot-WashUp  comment  is  documented  by  more  than  one  observer 
(as  is  often  the  case)  only  one  cross-reference  is  made  from  Section  6.  A  comment  called  “codel” 
that  is  the  same  as  comment  “code2”  and  thus  not  cross-reference  is  marked  as  follows:  codel  = 
code2  . 

Table  9  is  an  index  into  the  Hot  Wash-Up  Comments  pages. 


Table  9  Hot  Wash-Up  Comments 


Number 

Observer 

Time 

C.3.1.1 

Valerie 
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Vincent 

Nov  13  15:30 
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Mike 
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Bill 

C.3.2.1 

Vincent 

C.3.2.2 

Valerie 

Nov  14  10:30 

C.3.2.3 
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Bill 

C.3.3.1 
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Nov  14  13:00 
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Mike 

C.3.3.4 

Bill 

C. 3.4.1 
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C.3.4.2 
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Nov  14  14:30 

C.3.4.3 

Valerie 
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h  I 'yOUfU  c-~J  tiLd  f}rt^  U*>/f  ryr  l 

CU%  tc^  hr  4  y*Cf/k  /k 


C.3.3. 2 


C. 3. 3.3a 

■  l~W  TWjU - 4  L"#„. 

_ ^  ~|isl  -Wi^  fW  Xcfa  n  l%^U-  4^  tuvei-io  ,4. 

O'  . ^  \T . 

C. 3.3.3b 

» 

UJ  L-k  «~-ryJX^  .frW^  «.  1 .  Low  —  ■<_  VOS  ^d*.  4-<  *~J<Ll__a. 

O 

CO 

CO 

.3c 

“  e^irr^  1°  CW  /ivs  v\J~tLc 

_ 

C.3.3.3d 

kl<-«a4  ''a>Uw-»^Co>v'S  /“^.c(nCt>-<v5  -ev^/s 

C.3.3.3e 


- 4\n-«-bkg..'S  lo'iK 


^  Q.  ■  ls&U*d  e-ygjs  ? 


C.3.3.3f 


A  • ~1kf,  U~ 


>^vj-  ■ 


T-e^yi  b4^  °~»-  ^  •^jatuJ.Ci^ 


C.3.3.3g 


. U^.A  -HU. . <>  -  LaL  <S>v- 


i^V--^=V  eot^4>-j  _  cj-^  ^-te^FSi^V 


C.3.3.3h 


H-«wj 


e  c*^-  uh. 


srtt^J  JsiC^-C^-L  e^v-a^s  ^jia^/> 


. 5r . .  — lb  cfck  cj  Co 


C.3.3.3i 


..  tajri-w  v-.:  ct— .  sjLj-  '1  ■k^XtV1  ioL-^Is 

*.  l/^vG|  oc^jl  f  kjJ  irv-4~  *  ^  k_  S-ei 


C.3.3.3 


Exit  Interview  -  Group  Hotwash 


C. 3. 3.4a  =  C3.3  3c 


i  1 


What  are  your  general  impressions  of  how  this  type  of  functionality  might  impact  your  work? 

U(  n . <  I  V  . C.L  ^ 

(  (  i~  j.'  -j  I . 


What  were  the  most  useful  features/strengths  of  the  applications? 

'v  l  5  a  ;.M  j  i — J •" 


C.3.3.4b  =  C3.3.3d 


C.3.3.4c  =  C3.3.3e 


C.3.3.4d  =  C3.3.3g 


. . >H.~V 


1 


,J  /uv'  . f  iff 

t/n,  h 

W]  /  fi'  i’rfi  i  ‘  ,  p, 

^/'l  Llc-M  fU  /,nIw  (,  L  I  a 

What  were  the  biggest  roadblocks  in  completing  the  Carni&l  scenario’)  ,  Vv^‘  H 


v-OOvt 


n 


Do  you  have  any  thoughts  for  how  the  apps  could  be  improved? 


C.3.3.4e  =  C3.3.3a 


C.3.3.4f  =  C3.3.3b 


C.3.3.4g  =  C3.3.3h 


urtvU  Lw  fe  7Ka^  lV.  j 

4  ,3  ll>  (jv  U'  C^l 


tXv 


Any  other  thoughts? 


t'Hj  '| w cX,  i  vl\V -jl.  -p, 


/■■I — ti'. 


3  ^ 


,r.l 


C.3.3.4h  =  C3.3.3m 


/lA-  ^  IhLa .  /"  J?-* 


X 


n~~</y  t 


C. 3.3.4 


Exit  Interview  -  Group  Hotwash 


C.3.4.1a 


C.3.4.1b 


tionality  might  impact  your  work? 

'f^U  S  A(  fai,  /  -f^r  Ltv 

<>  '/  >  Lly^  . 


.  r. 1- . v-t  c-  *-s  j  j-J 


0.3.4.1c 


What  were  the  most  useful  features/strengths  of  the  applications? 

”  ^  yd  'c . . . /Lx.""-hV  ’  •  ! 

t  A-m 


C.3.4.1d 


S  d>  ^  cL-t  A 
1 


^  Jvv  '■ . *'  f*LP  S  ‘7  !  '-y)„>,S  J,  (A 


W'»n 


't  A-O  t>'W  ^  y_K_^ 


b  ci-o 


J 


0->'\ 


C.3.4.1e 


What  were  the  biggest  roadblocks  in  completing  the  Carnival  scenario? 

l/i'iK-'i  ii-'i7^(Vv)y\  tA-y>p  J  f.  ^  , 

<2J>\  .  4  VC  KM (  LXv 

| .  O  ■  . . -),’rt,'",  >v4A 

3V  ^^*79^  f::i  ivot . c; . <i  J...;  Akcy-cvi 


C. 3.4. If 


C.3.4.1g 


Do  you  have  any  thoughts  for  how  the  apps  could  be  improved? 

.  .  j  f 

jo  kvK  t  Uy*£  d/iu^w  W4}  ^tuyfa1 . f 

('■&1  \Ji  ^  V  f  vHA 


f<V 


C.3.4.1h 


i  Kef. . a  p1'  (Lit 

r  / 


&J(  m  (a  lit  v. 


C.3.4. 1 i 


f  ,-j  j  ' ) 

TV' t"  Oy1  u  A,;1  V  j  t.  (  y'  V  \  r-y  7'V  fyU  J  .  Y\ 

r*'^rl  . . lJ,v')  /I  ,  . {yc^L'  j>'A'3'  *j  Uiri, 

^ *^L~  h»j  wW, xj>  -It.  |  A-  / 

,  H  JK  nV  Ur4  IS 


C.3.4. 1j 


Any  other  thoughts? 

A  K pu  .  tatcA.  ‘v  ,}<_v  [j  i){ 

M.CC: .  4t  ><.: 


<  i- 

' . 1  y  U  O" 


I  A  C  ^  ^  7  i  v 

"V  M  SL, 


£>U> 


Ajfcsi pcy^;?  /'. 


i  ■  .  . 


^  j-* 


;Ou.->7  i 


/c  ;>x 


CL 


"W 


I'W  S 


!) 


W  w 


C.3.4. Ik 


. r  ik.C  \  <>u . -n.  J  -f 


3  ^'v  O 


12  Nov  2013  V  ^-3  ‘A...., 


~nV . Lt'Ut. 

^ ^  N.M'-ei A?  \> t  3/svy?Ua_ 

' — '?  cicry  (5^4 .  cAyAc  A  ,v  ^ 


4  4rK. 


C.3.4. 1 


C.3.4.2 


Any  other  thoughts? 


B.4  Filled-Out  System  Usability  Survey  Sheets 


The  following  pages  show  the  original  System  Usability  Sheets  that  were  collected  from  the 
participants.  Two  participants  did  not  fill  the  survey  in  the  last  session. 
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Original  System  Usability  Survey 


DEFENCE 


Eculus 


System  Usability  Survey 

Mark  one  box  in  each  row. 


Session  #: 


z 


Participant  #: 


Strongly 

disagree 

Strongly 

agree 

1 .  1  think  that  1  would  like  to  use  these  apps 
frequently 

1  2  3 

4  s 

2.  I  found  the  apps  unnecessarily  complex 

-  t/ 

1  2  3 

4  5 

3.  I  thought  the  apps  were  easy  to  use 

IT 

1  2  3 

4  5 

4.  1  think  that  I  would  need  the  support  of  a 
technical  person  to  be  able  to  use  these  apps 

</ 

1  2  3 

4  5 

5.  1  found  the  various  functions  in  the  apps 
were  well  integrated 

1  2  3 

_ / _ 

4  5 

6.  1  thought  there  was  too  much  inconsistency 
in  the  apps 

V 

1  2  3 

4  5 

7.  I  would  imagine  that  most  people  would 
learn  to  use  the  apps  very  quickly 

v r~ 

1  2  3 

4  5 

8.  I  found  the  apps  very  awkward  to  use 

J 

1  2  3 

4  5 

9.  1  felt  very  confident  using  the  apps 

1  2  3 

4  5 

10. 1  needed  to  learn  a  lot  of  things  before  1 
could  get  going  with  these  apps 

yf 

1  2  3 

4  5 

12  Nov  2013 


(<Jo  J 


Hr 


f 


p) 


L 

Eculus 


System  Usability  Survey  Session#:^!  Participant#:^ 


Mark  one  box  in  each  row. 


1.  I  think  that  I  would  like  to  use  these  apps 
frequently 


2.  1  found  the  apps  unnecessarily  complex 


3.  1  thought  the  apps  were  easy  to  use 


4.  I  think  that  J  would  need  the  support  of  a 
technical  person  to  be  able  to  use  these  apps 


5.  1  found  the  various  functions  in  the  apps 
were  well  integrated 


6.  1  thought  there  was  too  much  inconsistency 
in  the  apps 


7.  1  would  imagine  that  most  people  would 
learn  to  use  the  apps  very  quickly 


8.  I  found  the  apps  very  awkward  to  use 


9.  1  felt  very  confident  using  the  apps 


10. 1  needed  to  learn  a  lot  of  things  before  I 
could  get  going  with  these  apps 


Strongly 

disagree 


Strongly 

agree 


LX 

1  2  3 

-  ^  . . . . 

4  5 

1  2  3 

4  5 

y 

17 

1  2  3 

— .  

4  5 

1/ 

1  2  3 

4  5 

12  3 

- .Z _ 

4  5 

V-/ 

1  2  3 

4  5 

1  2  3 

4  5 

l/ 

1 

1  2  3 

4  5 

\7 

1  2  3 

- - _ 

4  5 

2  3  4  5 


12  Nov  2013 


OEVHNCB  I  ijy  DtPBNSB 


Bcuius 


System  Usability  Survey  Session*^  Participant*^ 
Mark  one  box  in  each  row. 


1 .  I  think  that  1  would  like  to  use  these  apps 
frequently 

2.  1  found  the  apps  unnecessarily  complex 


3.  1  thought  the  apps  were  easy  to  use 


4.  I  think  that  1  would  need  the  support  of  a 
technical  person  to  be  able  to  use  these  apps 


5.  I  found  die  various  functions  in  the  apps 
were  well  integrated 


6.  1  thought  there  was  too  much  inconsistency 
in  the  apps 


7.  1  would  imagine  that  most  people  would 
learn  to  use  the  apps  very  quickly 


8.  I  found  the  apps  very  awkward  to  use 


9.  1  felt  very  confident  using  the  apps 


10. 1  needed  to  learn  a  lot  of  things  before  I 
could  get  going  with  these  apps 


Strongly 

disagree 

Strongly 

agree 

1  2  3 

4  5 

1  ✓ 

1 

1  2  3 

4  5 

/ 

I 

L 

t/ 

1 

1  2  3 

_ _ L _ 

4  5 

1  2  3 

4  5 

1 

1  2  3 

- L _ 

4  5 

V 

1  2  3 

4  3 

1 

J  2  3 

_ _ L _ 

4  5 

1  2  3 

4  5 

/ 

_ 

’/  *  1 

4  5 

1  2  3 

4  s 

12  Nov  2013 


Eculus 


■RL)''* 


00 

System  Usability  Survey 

Mark  one  box  in  each  row. 


n 


\° 


^  9 


Session  #: 


Participant  #: _ 


I .  I  think  that  I  would  like  to  use  these  apps 
frequently 


2.  I  found  the  apps  unnecessarily  complex 

(  I 'fjnfv  /Jj*  -f.n;  (-4  c 

AMc  c/.ct  h  ymP  ** 

Mc  .'cM.r  He  dJer 

j.  I  thought  the  apps  were  easy  to  use 

t  ac/J  c.  el\k  /o  c  /'•Mi 1 

f-0  -JVr  fufhef  he* 

fe  ?«•*/•«./>  //>  d^f-t. 

4.  1  think  that  I  would  need  the  support  of  a 
technical  person  to  be  able  to  use  these  apps 


5.  1  found  the  various  functions  in  the  apps 
were  well  integrated 


6.  1  thought  there  was  too  much  inconsistency 
in  the  apps 


7.  1  would  imagine  that  most  people  would 
learn  to  use  the  apps  very  quickly 


8.  I  found  the  apps  very  awkward  to  use 


9.  I  felt  very  confident  using  the  apps 


10. 1  needed  to  learn  a  lot  of  things  before  i 
could  get  going  with  these  apps 

/  >  O  -\/»  ^  40cJ  /*'#*■"  ^ 


Strongly 

disagree 

Strongly 

agree 

1  2  6J 

4  5 

1  U  3 

4  5 

1 

1  2  6J 

4  5 

(9  2  3 

4  5 

1  2  3 

(iJy  5 

L 

(Jf  2 

4  5 

1  2  3 

(9  5 

1  &  3 

4  5 

1  2  3 

(if  3 

1  Q  3 

4  5 

M'/iL  >‘tf  of  / f  a y<. 

M'S  Con/J  futvffnh 
]j£>  , 


12  Nov  2013 


5 

Eculus 


DBKkNCB 


yJo 0  ^ 

1° 

System  Usability  Survey 


Q 


Session  #: 


2 


Participant  #: 


Mark  one  box  in  each  row. 


1 .  I  think  that  I  would  like  to  use  these  apps 
frequently 


2.  I  found  the  apps  unnecessarily  complex 


3.  1  thought  the  apps  were  easy  to  use 


4.  !  think  that  I  would  need  the  support  of  a 
technical  person  to  be  able  to  use  these  apps 


5.  I  found  the  various  functions  in  the  apps 
were  well  integrated 


6.  1  thought  there  was  too  much  inconsistency 
in  the  apps 


7.  1  would  imagine  that  most  people  W'ould 
learn  to  use  the  apps  very  quickly 


8.  1  found  the  apps  very  awkward  to  use 


9.  1  felt  very  confident  using  the  apps 


10. 1  needed  to  learn  a  lot  of  things  before  I 
could  get  going  with  these  apps 


12  Nov  201 3 


DEFENCE 


no'- 


System  Usability  Survey 

Mark  one  box  in  each  row. 


Session  #:. 


u 

Eculus 

I 


Participant  #: 


1.  i  think  that  I  would  like  to  use  these  apps 
frequently 


2.  I  found  the  apps  unnecessarily  complex 


3.  I  thought  the  apps  were  easy  to  use 


4.  I  think  that  1  would  need  the  support  of  a 
technical  person  to  be  able  to  use  these  apps 


5.  I  found  the  various  functions  in  the  apps 
were  well  integrated 


6.  I  thought  there  was  too  much  inconsistency 
in  the  apps 


7.  1  would  imagine  that  most  people  would 
leam  to  use  the  apps  very  quickly 


8.  1  found  the  apps  very  awkward  to  use 


9.  1  felt  very  confident  using  the  apps 


1 0. 1  needed  to  leant  a  lot  of  things  before  I 
could  get  going  with  these  apps 


Strongly 

disagree 


Strongly 

agree 


\( 

1  2  3 

4  5 

x. 

1 

2  3 

4  5 

X 

1  2  3 

4  5 

|  "'X 

1  2  3 

4  5 

XJ 

1  2  3 

4  5 

I  ^ 

1 

1  2  3 

4  5 

XI 

1 

1  2  3 

4  5 

[ 

X 

1  2  3 

4  5 

V 

X/ 

1  2  3 

4  5 

x» 

1  2  3 

4  5 

12  Nov  2013 


RD;‘ 


/\V  If  ! 


Bculus 


System  Usability  Survey 


Mark  one  box  in  each  row. 


Session  #: 


Participant  #. 


1 .  1  think  that  I  would  like  to  use  these  apps 
frequently 


2.  1  found  the  apps  unnecessarily  complex 


3.  1  thought  the  apps  were  easy  to  use 


4.  1  think  that  1  would  need  the  support  of  a 
technical  person  to  be  able  to  use  these  apps 


5.  1  found  the  various  functions  in  the  apps 
were  well  integrated 


6.  1  thought  there  was  too  much  inconsistency 
in  tire  apps 


7.  1  would  imagine  that  most  people  would 
learn  to  use  the  apps  very  quickly 


8.  I  found  the  apps  very  awkward  to  use 


9.  I  felt  very  confident  using  the  apps 


10. 1  needed  to  learn  a  lot  of  things  before  I 
could  get  going  with  these  apps 


Strongly 

disagree 

Strongly 

agree 

X 

1  2  3 

4  5 

X 

1  2  3 

4  5 

X 

1  2  3 

4  5 

X 

1  2  3 

4  5 

X 

1  2  3 

4  5 

1  2  3 

4  5 

X. 

1  2  3 

4  5 

1  2  3 

4  5 

r  2  3 

4  5 

X 

l  2  3 

4  5 

12  Nov  2013 


Bculus 


fa  v'  H-  /  w 

System  Usebility  Sun/ey  Session#: _  Participant#: 

Mark  one  box  in  each  row. 


1 .  I  think  that  I  would  like  to  use  these  apps 
frequently 


2.  I  found  the  apps  unnecessarily  complex 


3.  1  thought  the  apps  were  easy  to  use 


4.  1  think  that  1  would  need  the  support  of  a 
technical  person  to  be  able  to  use  these  apps 


5.  I  found  the  various  functions  in  the  apps 
were  well  integrated 


6.  I  thought  there  was  too  much  inconsistency 
in  the  apps 


7,  I  would  imagine  that  most  people  would 
learn  to  use  the  apps  very  quickly 


8.  I  found  the  apps  very  awkward  to  use 


9.  I  felt  very  confident  using  the  apps 


10. 1  needed  to  learn  a  lot  of  things  before  I 
could  get  going  with  these  apps 


Strongly  Strongly 

disagree  agree 


12  Nov  2013 


3 

Ecu  I IJS 


System  Usebility  Survey  session#: _  participant#: 

Mark  one  box  in  each  row. 


1 .  1  think  that  I  would  like  to  use  these  apps 
frequently 


2.  I  found  the  apps  unnecessarily  complex 


3.  1  thought  the  apps  were  easy  to  use 


4.  1  think  that  I  would  need  the  support  of  a 
technical  person  to  be  able  to  use  these  apps 


5.  I  found  the  various  functions  in  the  apps 
were  well  integrated 


6.  1  thought  there  was  too  much  inconsistency 
in  the  apps 


7.  I  would  imagine  that  most  people  would 
learn  to  use  the  apps  very  quickly 


8.  I  found  the  apps  very  awkward  to  use 


9.  I  felt  very  confident  using  the  apps 


10. 1  needed  to  learn  a  lot  of  things  before  I 
could  get  going  with  these  apps 


Strongly 

disagree 

Strongly 

agree 

X 

1  2  3 

4  5 

_x 

1  2  3 

4  5 

X 

1  2  3 

4  5 

_x 

1  2  3 

4  5 

X 

1 

1  2  3 

4  5 

1  x 

1  2  3 

4  3 

X 

1  2  3 

4  5 

X 

1  2  3 

4  5 

X 

1  2  3 

4  5 

-  x 

1  2  3 

4  5 

12  Nov  2013 


\0 


DEFENCE 


\3  KcJ 


Eculus 


System  Usebility  SUPi/ey  Session#: _  Participant#: 

Mark  one  box  in  each  row. 


1 .  I  think  that  I  would  like  to  use  these  apps 
frequently 


2.  1  found  the  apps  unnecessarily  complex 


3.  1  thought  the  apps  were  easy  to  use 


4.  I  think  that  I  would  need  the  support  of  a 
technical  person  to  be  able  to  use  these  apps 


5.  1  found  the  various  functions  in  the  apps 
were  well  integrated 


6.  1  thought  there  was  too  much  inconsistency 
in  the  apps 


7.  I  would  imagine  that  most  people  would 
learn  to  use  the  apps  very  quickly 


8.  1  found  the  apps  very  awkward  to  use 


9.  I  felt  very  confident  using  the  apps 


10. 1  needed  to  learn  a  lot  of  things  before  I 
could  get  going  with  these  apps 


Strongly 

disagree 

Strongly 

agree 

1  2  3 

4  5 

X 

1  2  3 

4  5 

1  2  3 

4  5 

1 

1  2  3 

4  5 

> 

A 

1  2  3 

4  5 

1  2  3 

4  5 

1  2  3 

4  5 

_ ; 

1  2  3 

4  5 

1  2  3 

4  5 

1  2  3 

4  5 

12  Nov  2013 


\3, 


“cuius 


RD- 


System  Usability  Survey 

Mark  one  box  in  each  row. 


Session  #: 


Participant  #: 


1.  I  think  that  I  would  like  to  use  these  apps 
frequently 


2.  1  found  the  apps  unnecessarily  complex 


3.  1  thought  the  apps  were  easy  to  use 


4.  1  think  that  1  would  need  the  support  of  a 
technical  person  to  be  able  to  use  these  apps 


5.  1  found  the  various  functions  in  the  apps 
were  well  integrated 


6.  I  thought  there  was  too  much  inconsistency 
in  the  apps 


7.  I  would  imagine  that  most  people  would 
learn  to  use  the  apps  very  quickly 


8.  I  found  the  apps  very  awkward  to  use 


9.  I  felt  very  confident  using  the  apps 


10. 1  needed  to  learn  a  lot  of  things  before  1 
could  get  going  with  these  apps 


Strongly 

disagree 

Strongly 

agree 

1  2  3 

4  3 

1  2  3 

4  5 

| 

1  2  3 

4 

5 

1  2  3 

4  5 

1  2  3 

4  5 

1  2  3 

4  5 

/ 

r~>4 

1  2  3 

4  5 

1  2  3 

4  3 

1  2  3 

4  5 

P< 

1  2  3 

4  5 

12  Nov  2013 


!Z_ 


DF.FENCB 


sJo  v/ 


Bculus 


System  Usebility  Survey  Session#: _  Participant#: 

Mark  one  box  in  each  row. 


1 .  1  think  that  I  would  like  to  use  these  apps 
frequently 

2.  1  found  the  apps  unnecessarily  complex 

Strongly 

disagree 

Strongly 

agree 

1  2  3 

4  5 

3.  I  thought  the  apps  were  easy  to  use 
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5.  1  found  the  various  functions  in  the  apps 

6.  I  thought  there  was  too  much  inconsistency 

!  2  3 

4  5 

7.  1  would  imagine  that  most  people  would 
learn  to  use  the  apps  very  quickly 

1  2  3 

4  S 

1  2  3 

4  5 

8.  1  found  the  apps  very  awkward  to  use 

1  2  3 

4  5 

9.  1  felt  very  confident  using  the  apps 

1  2  3 

4  5 

10. 1  needed  to  learn  a  lot  of  things  before  1 
could  get  going  with  these  apps 
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List  of  symbols/abbreviations/acronyms/initialisms 


App 

Application  Software,  also:  Widget 

ARP 

Advanced  Research  Project 

ASM 

Analysis  Set  Manager 

CBS  A 

Canada  Border  Services  Agency 

CCG 

Canadian  Coast  Guard 

CORA 

Centre  for  Operational  Research  and  Analysis 

DFO 

Department  of  Fisheries  and  Oceans 

DRDC 

Defence  Research  &  Development  Canada 

GPW 

Global  Positioning  Warehouse 

GUI 

Graphical  User  Interface 

IRCS 

International  Radio  Call  Sign 

ISTIP 

Intelligence  Science  and  Technology  Integration  Platform 

IT 

Information  Technology 

LCdr 

Lieutenant  Commander 

MDA 

Maritime  Domain  Awareness 

MG 

Magnets  Grid 

MS 

Microsoft 

MSOC 

Marine  Security  Operations  Centre 

MVAP 

Maritime  Visual  Analytic  Prototype 

NATO 

North  Atlantic  Treaty  Organization 

PAVM 

Prototype  d’ Analyse  Visuelle  Maritime 

POI 

Person  of  Interest 

RCMP 

Royal  Canadian  Mounted  Police 

RCN 

Royal  Canadian  Navy 

RMP 

Recognized  Maritime  Picture 

ROI 

Region  of  Interest 

sus 

System  Usability  Scale 

VA 

Visual  Analytics 

VOI 

Vessel  of  Interest 

VTS 

Vessel  Tracking  System 

WAS 

Widget  Application  Shell 

55 


